
Most of my career was iOS. I shipped apps, lived in Xcode, and got very used to the nice, tight feedback loop of building for one platform. At Facebook (now Meta), I expanded that into working with custom frameworks and tools outside the typical iOS development workflow. Later I moved into management and leadership work, which meant less hands-on coding day to day. Everything else (backend, frontend, databases, etc.) mostly happened on evenings and weekends as side projects. After I left Sentry, that weekend-and-evening hacking turned into full-time exploration.

As part of my exploration of LLMs and related infrastructure, I inevitably came across tool calling.

I’ve been building a workout manager and AI workout coach for myself ([fitness.antlab.ai](https://fitness.antlab.ai)). This gateway work grew out of my own need: I want models to take actions safely and predictably, not just generate text confidently. More on the AI coach another time. I think it’s a cool product in itself, and it’s gotten really good feedback from a few professional and amateur athlete friends who’ve been using it.

## Why build a gateway?

Tool calling is the part that makes LLMs useful beyond chat. Instead of guessing, the model can fetch data or run an action. The problem is that every provider implements “tool calling” with slightly different semantics, and wiring tools plus OAuth into every backend service becomes a mess quickly.

A gateway gave me three practical wins:

- Centralised configuration: define tools once, expose them to different models.
- Authentication and permissions: handle OAuth and per-tool scopes in one place.
- Consistency: normalise provider responses so callers do not need a pile of special cases.

My longer-term goal was more ambitious: manage context in the gateway and eventually stop depending on MCP servers. Even before that, the edge cases taught me where the spec ends and reality begins.

## Edge cases and implementation lessons

### 1. Resource metadata discovery is not consistent

Different MCP servers expose OAuth resource metadata in different places. What ended up working in practice was checking three locations, in this order:

1. The resource URL itself
2. `/.well-known/oauth-protected-resource` appended to the resource URL
3. `/.well-known/oauth-protected-resource` at the host level

This is one of those “I wish it was simpler” parts. You cannot assume everyone follows the same discovery endpoint.

### 2. `WWW-Authenticate` parsing is easy to botch

Some providers return a 401 with a `WWW-Authenticate` header that includes a `resource_metadata` URL. Parsing that header is fiddly, and it is very easy to accept something malformed (or reject something valid).

In my implementation there is still a TODO to add proper unit tests for this case, because it is exactly the kind of bug that only shows up in production.

### 3. Specs leave room, and providers take it

There are a few validation checks I still consider “must do”, but they take time:

- Supported endpoint auth methods
- Supported response types
- Supported grant types
- Refresh token support

The spec leaves room for interpretation, and providers use that freedom.

### 4. Discovery can fail even when you have credentials

One pragmatic change that improved UX: if discovery fails but the gateway has preconfigured OAuth credentials (some providers need this), it still populates client ID/secret so users can try authenticating. Some providers have flaky metadata endpoints, and letting the user proceed is better than a hard stop.

### 5. Token refresh needs a bit of slack

Tokens expiring “right now” is a common failure mode. I added a 30-second skew so the gateway refreshes slightly before expiry and avoids race conditions.

## Termination reasons: normalised vs native

Models report “why I stopped generating” in different ways. The gateway tracks two fields:

- `finishReason`: a normalised reason (for example `stop`, `tool_calls`, `length`)
- `nativeFinishReason`: the provider’s raw reason (for example `STOP` or `MAX_TOKENS`)

Here’s a simplified version of the check that decides whether to warn the caller about an upstream failure:

```go
func shouldNotifyUpstreamFailure(term upstreamTermination) bool {
    // Notify if nativeFinishReason indicates an error (non-empty and not "stop" or "max_tokens" - case insensitive)
    if term.nativeFinishReason != "" {
        native := strings.ToLower(term.nativeFinishReason)
        if native != "stop" && native != "max_tokens" && native != "function_call" && native != "tool_calls" {
            return true
        }
    }

    // Notify if finishReason indicates an error (not "stop", "tool_calls", or "length")
    if term.finishReason != "" {
        reason := strings.ToLower(term.finishReason)
        if reason != "stop" && reason != "tool_calls" && reason != "length" {
            return true
        }
    }
    return false
}
```

Two patterns that mattered:

- Gemini tends to use uppercase stop codes like `STOP` and `MAX_TOKENS`, while others use lowercase. Lowercasing avoids false alarms.
- Some APIs use `tool_calls`, others use `function_call`. In both cases it usually means “the model successfully decided to call a tool”, so I treat them the same.

Anything else (for example `content_filter`) triggers a warning, because it often means the response did not complete normally.

## How models differ in tool calling

The uncomfortable truth is that tool calling is learned behaviour. The API surface looks similar, but models make different choices about when to call tools, how strict they are about schemas, and how often they “give up” and answer in plain text.

### OpenAI (ChatGPT, GPT-4 class models)

OpenAI gives you explicit control over tool choice (auto, required, none). In practice, I’ve found behaviour varies a lot by model and prompt style. Some models will happily call a tool early, others need a very clear nudge, and the same toolset can behave differently across versions.

What helped me most was keeping tool names and parameter descriptions short, and not exposing too many tools at once.

### Anthropic (Claude)

Claude is generally comfortable mixing tool use with text, and it can chain tool calls naturally. It can also be more proactive than you expect, which is great when it is right and annoying when it is not. I ended up tuning tool descriptions more carefully to reduce “helpfulness” that turned into unnecessary tool calls.

### Google (Gemini)

Gemini has a lot of surface area and a lot of knobs, and when it works it can be excellent. The reliability was the harder part for me. I’ve seen runs where it calls tools correctly, then later in the day it answers in plain text instead of invoking the same tool, or it flips between behaviours across requests. That unpredictability is what pushed me to invest in stricter monitoring around termination reasons and tool-calling outcomes.

## Takeaways

Tool calling is not a pure spec implementation. It is model behaviour. If you want it to feel consistent to users, you have to do the boring work:

- Design tools for clarity. Short names, tight schemas, good descriptions.
- Do not expose everything at once. Too many tools lowers accuracy.
- Normalise and monitor. Treat unexpected termination reasons as real signals, not noise.
- Test across models. The same prompt and toolset will behave differently.

## Future work (and a note on the AI coach)

This gateway is still very much iterative. The next things I want to tackle:

- Better tool definitions, with examples that steer models toward correct calls.
- Lightweight JS tools so users can define simple functions without running a full MCP server.
- Automated regression tests that run prompts across models to catch behaviour drift.

And yes, the other project is still going: Fitter is a structured workout manager and AI coach. It lets athletes plan sessions, generate intervals, and sync workouts to devices. The coach side is where tool calling really earns its keep, but that’s a separate post.

## Conclusion

Tool calling is what turns LLMs into something you can build a product around, but it gets messy fast across providers. Finish reasons are inconsistent, tool-calling semantics vary, and reliability can swing from model to model. A gateway helps, but only if you treat standards as guidelines, normalise responses aggressively, and keep tightening your tool definitions based on real behaviour.


Jernej Štrasner

Building an LLM Gateway: Lessons from Tool Calling Across Models