
Most of my career was iOS. I shipped apps, lived in Xcode, and got very used to the nice, tight feedback loop of building for one platform. At Facebook (now Meta), I expanded that into working with custom frameworks and tools outside the typical iOS development workflow. Later I moved into management and leadership work, which meant less hands-on coding day to day. Everything else (backend, frontend, databases, etc.) mostly happened on evenings and weekends as side projects. After I left Sentry, that weekend-and-evening hacking turned into full-time exploration.

As part of my exploration of LLMs and related infrastructure, I inevitably came across tool calling.

I’ve been building a workout manager and AI workout coach for myself ([fitness.antlab.ai](https://fitness.antlab.ai)). This gateway work grew out of my own need: I want models to take actions safely and predictably, not just generate text confidently. More on the AI coach another time. I think it’s a cool product in itself, and it’s gotten really good feedback from a few professional and amateur athlete friends who’ve been using it.

## Why build a gateway?

Tool calling is the part that makes LLMs useful beyond chat. Instead of guessing, the model can fetch data or run an action. The problem is that every provider implements “tool calling” with slightly different semantics, and wiring tools plus OAuth into every backend service becomes a mess quickly.

A gateway gave me three practical wins:

- Centralised configuration: define tools once, expose them to different models.
- Authentication and permissions: handle OAuth and per-tool scopes in one place.
- Consistency: normalise provider responses so callers do not need a pile of special cases.

My longer-term goal was more ambitious: manage context in the gateway and eventually stop depending on MCP servers. Even before that, the edge cases taught me where the spec ends and reality begins.

## Edge cases and implementation lessons

### 1. Resource metadata discovery is not consistent

Different MCP servers expose OAuth resource metadata in different places. What ended up working in practice was checking three locations, in this order:

1. The resource URL itself
2. `/.well-known/oauth-protected-resource` appended to the resource URL
3. `/.well-known/oauth-protected-resource` at the host level

This is one of those “I wish it was simpler” parts. You cannot assume everyone follows the same discovery endpoint.

### 2. `WWW-Authenticate` parsing is easy to botch

Some providers return a 401 with a `WWW-Authenticate` header that includes a `resource_metadata` URL. Parsing that header is fiddly, and it is very easy to accept something malformed (or reject something valid).

In my implementation there is still a TODO to add proper unit tests for this case, because it is exactly the kind of bug that only shows up in production.

### 3. Specs leave room, and providers take it

There are a few validation checks I still consider “must do”, but they take time:

- Supported endpoint auth methods
- Supported response types
- Supported grant types
- Refresh token support

The spec leaves room for interpretation, and providers use that freedom.

### 4. Discovery can fail even when you have credentials

One pragmatic change that improved UX: if discovery fails but the gateway has preconfigured OAuth credentials (some providers need this), it still populates client ID/secret so users can try authenticating. Some providers have flaky metadata endpoints, and letting the user proceed is better than a hard stop.

### 5. Token refresh needs a bit of slack

Tokens expiring “right now” is a common failure mode. I added a 30-second skew so the gateway refreshes slightly before expiry and avoids race conditions.

## Termination reasons: normalised vs native

Models report “why I stopped generating” in different ways. The gateway tracks two fields:

- `finishReason`: a normalised reason (for example `stop`, `tool_calls`, `length`)
- `nativeFinishReason`: the provider’s raw reason (for example `STOP` or `MAX_TOKENS`)

Here’s a simplified version of the check that decides whether to warn the caller about an upstream failure:

```go
func shouldNotifyUpstreamFailure(term upstreamTermination) bool {
    // Notify if nativeFinishReason indicates an error (non-empty and not "stop" or "max_tokens" - case insensitive)
    if term.nativeFinishReason != "" {
        native := strings.ToLower(term.nativeFinishReason)
        if native != "stop" && native != "max_tokens" && native != "function_call" && native != "tool_calls" {
            return true
        }
    }

    // Notify if finishReason indicates an error (not "stop", "tool_calls", or "length")
    if term.finishReason != "" {
        reason := strings.ToLower(term.finishReason)
        if reason != "stop" && reason != "tool_calls" && reason != "length" {
            return true
        }
    }
    return false
}
```

Two patterns that mattered:

- Gemini tends to use uppercase stop codes like `STOP` and `MAX_TOKENS`, while others use lowercase. Lowercasing avoids false alarms.
- Some APIs use `tool_calls`, others use `function_call`. In both cases it usually means “the model successfully decided to call a tool”, so I treat them the same.

Anything else (for example `content_filter`) triggers a warning, because it often means the response did not complete normally.

## How models differ in tool calling

The uncomfortable truth is that tool calling is learned behaviour. The API surface looks similar, but models make different choices about when to call tools, how strict they are about schemas, and how often they “give up” and answer in plain text.

### OpenAI (ChatGPT, GPT-4 class models)

OpenAI gives you explicit control over tool choice (auto, required, none). In practice, I’ve found behaviour varies a lot by model and prompt style. Some models will happily call a tool early, others need a very clear nudge, and the same toolset can behave differently across versions.

What helped me most was keeping tool names and parameter descriptions short, and not exposing too many tools at once.

### Anthropic (Claude)

Claude is generally comfortable mixing tool use with text, and it can chain tool calls naturally. It can also be more proactive than you expect, which is great when it is right and annoying when it is not. I ended up tuning tool descriptions more carefully to reduce “helpfulness” that turned into unnecessary tool calls.

### Google (Gemini)

Gemini has a lot of surface area and a lot of knobs, and when it works it can be excellent. The reliability was the harder part for me. I’ve seen runs where it calls tools correctly, then later in the day it answers in plain text instead of invoking the same tool, or it flips between behaviours across requests. That unpredictability is what pushed me to invest in stricter monitoring around termination reasons and tool-calling outcomes.

## Takeaways

Tool calling is not a pure spec implementation. It is model behaviour. If you want it to feel consistent to users, you have to do the boring work:

- Design tools for clarity. Short names, tight schemas, good descriptions.
- Do not expose everything at once. Too many tools lowers accuracy.
- Normalise and monitor. Treat unexpected termination reasons as real signals, not noise.
- Test across models. The same prompt and toolset will behave differently.

## Future work (and a note on the AI coach)

This gateway is still very much iterative. The next things I want to tackle:

- Better tool definitions, with examples that steer models toward correct calls.
- Lightweight JS tools so users can define simple functions without running a full MCP server.
- Automated regression tests that run prompts across models to catch behaviour drift.

And yes, the other project is still going: Fitter is a structured workout manager and AI coach. It lets athletes plan sessions, generate intervals, and sync workouts to devices. The coach side is where tool calling really earns its keep, but that’s a separate post.

## Conclusion

Tool calling is what turns LLMs into something you can build a product around, but it gets messy fast across providers. Finish reasons are inconsistent, tool-calling semantics vary, and reliability can swing from model to model. A gateway helps, but only if you treat standards as guidelines, normalise responses aggressively, and keep tightening your tool definitions based on real behaviour.


Building an LLM Gateway: Lessons from Tool Calling Across Models


I’ve been building products and tools for years — mostly iOS, lots of scripting, some backend, a lot of “make it work and ship it.” Recently I started poking into the AI space. Not just “call an LLM and magic happens,” but the *internals*. I want to understand enough to eventually build something serious on top of this tech.

One of the first things I kept hearing about in RAG (retrieval-augmented generation) setups was **chunking** and **embeddings** — and how they all tie into a **vector database**. I figured I’d document my beginner mental model.

## Step 1: Chunking
This part made sense quickly. You have long documents, but your embedding model (and later the LLM) has token limits. So you split the text into smaller **chunks**.

- Typical chunk size: ~500–800 tokens (2,000–3,200 characters in English)  
- Overlap: 10–20% so context flows between chunks  
- Too big = you get noisy, unfocused vectors. Too small = you lose context.

I now think of chunking as just a preprocessing step to keep everything bite-sized for the next phase.

## Step 2: Embeddings
Once you have chunks, you pass each one through an **embedding model**. This is where my “OK, but how?” curiosity kicked in.

The model outputs a **vector** — literally just an array of floats, like: [0.0213, -0.1342, 0.0876, …]

The magic is that similar meanings end up with vectors that are close together in this high-dimensional space. “Car” and “Automobile” will be neighbors. “Car” and “Banana” will be far apart.

It’s nothing like Levenshtein distance (string edits). Instead, embeddings are trained so semantic closeness translates to geometric closeness.

## Step 3: Storing in a Vector DB
Each chunk’s vector goes into a **row** in the vector database, alongside the original text and metadata. The DB builds an index so later we can search fast.

The vector DB doesn’t care about tokens or text anymore — it’s just arrays of numbers and math.

## Step 4: Retrieval
Here’s the clever bit:

1. User asks a question.  
2. You embed the question into its own vector (same model).  
3. The vector DB finds the **nearest neighbors** to that query vector.  
4. You fetch the corresponding chunks and feed them into your LLM for the final answer.

The naive way is to just scan all rows and pick the highest cosine similarity score (cosine similarity = dot product normalized for vector length). That works for small data, but with millions of vectors it’s too slow.

## Step 5: How Lookup is Actually Fast
Most vector DBs use an **approximate nearest neighbor** index — HNSW is the popular one. My mental model:

- Imagine all your points as cities on a map.  
- The DB builds a set of “highway” layers to jump across the space quickly, then “street” layers to refine locally.  
- When searching, it hops across highways, drops down to streets, and zeroes in on the closest points without visiting every “city.”

This gets you *almost* the same result as scanning everything, but way faster.

## Dot Product & Cosine Similarity (The Core Math)
The dot product is just multiplying each vector component, summing it up.  
Cosine similarity is the dot product divided by the product of vector lengths — which basically strips out magnitude and keeps only the **angle** between the vectors.

For embeddings, angle matters more than length, because we care about **direction in semantic space**, not absolute size.

## Final Mental Picture
- Chunk → embed → store in vector DB → embed query → find nearest vectors → send chunks to LLM.  
- Embeddings are just numbers.  
- Vector search is just “find points closest to this one” with clever shortcuts.  
- The rest is plumbing and product thinking.

I’m only scratching the surface, but already this feels a lot less like magic and more like a set of tools I can reason about.

Learning How Chunking, Embeddings, and Vector Databases Actually Work


As a (bored? topic for another post) startup founder currently between ventures, I've been methodically evaluating AI development tools to understand their practical applications and limitations. After extensive testing, particularly with Cursor and Claude Code, the results vary significantly across different technology stacks.

## The iOS Reality Check

My primary background is iOS development, where I've spent years building native Swift applications. AI tools in this domain remain frustratingly limited. The Apple ecosystem's complexity—from UIKit intricacies to SwiftUI's evolving patterns, Xcode's quirks, and platform-specific constraints—proves challenging for current AI models. While AI can generate basic Swift code, it struggles with the nuanced understanding required for production iOS development. It often writes outdated code, particularly when it comes to Swift 6 and concurrency. It can quickly iterate itself into a downward spirall of overcomplicated code which can be a complete mess. The feedback loops are slow, and the tools lack deep integration with Xcode workflows that iOS engineers rely on daily.

I've experimented with MCP implementations for Xcode, but the integration remains rudimentary compared to what's needed for serious iOS work. Claude often calls the tool wrong, tries a few times, and only then gets it right. This can be somewhat mitigated by having a well written CLAUDE.md file but it's not perfect.

## Web and Backend: A Different Story

The contrast with web development is stark. When working outside my primary domain—web frontend and backend development—AI tools prove remarkably effective. I recently built a complete [web version](https:///www.swimtemps.com) of my iOS app Swim using pretty much exclusively Claude Code. What would have taken weeks of research, boilerplate generation, and stack decisions was completed in a fraction of the time.

For Rust backend development, similar results emerge. The AI handles dependency management, error handling patterns, and API design with competence that approaches what I'd expect from a junior engineer familiar with the stack. However, due to Rust's intricacies it will still get things wrong. Armin Ronacher [recently wrote](https://lucumr.pocoo.org/2025/6/12/agentic-coding/) about LLMs being quite good at Go so I will give that a try next (and coincidentally, we chose Go as the primary backend stack at Specto for similar reasons why it works well with LLMs).

## Interaction Paradigm

The most significant shift isn't the code generation itself, but the interaction model. Rather than wrestling with command-line syntax or documentation, I can communicate with AI as I would with an engineer on my team. This conversational interface for system administration, project setup, and routine development tasks represents a meaningful productivity improvement. I generally run up to 3 terminal tabs with Claude in each one, either working on a different codebase/repository or even a different checkout of the same one.

I've been experimenting with bypassing permission prompts to increase iteration speed—a calculated risk that requires robust backup systems but dramatically improves the development flow. The AI maintains context across tasks, enabling complex multi-step operations that would typically require careful orchestration.

Claude Code is much easier to interact with in this new paradigm than Cursor. It lives in the terminal, focuses purely on the agentic aspect and doesn't even provide an editor. Cursor, as a vscode fork, is much more complicated and has a ton of options. Now ironically this post is being written in Cursor with a markdown preview plugin so there's still place for editors but I think Claude Code is on a better path.

## MCP and Automation

Beyond code generation, I've tested MCP implementations for browser automation with Puppeteer. The ability to describe user interface interactions in natural language and have them translated to reliable automation scripts opens interesting possibilities for testing and data collection workflows. This works pretty well most of the time but again, a well written CLAUDE.md file works wonders. As mentioned above, this works to some extent with Xcode as well but it's much more cumbersome. It will be interesting to follow the progress there. I've also tried using an MCP server for Railway (which is where I host most of my projects) and Vercel. For both I found it much easier to just get Claude to use the CLI tools for which I provided basic instructions in the respective CLAUDE.md files. The contents are basically the outputs of `vercel --help` and `railway --help`, condensed and with instructions to call the tool and command with `--help` to learn more when something doesn't work. Work wonders.

## Assessment

The current state suggests AI development tools are most valuable when working outside your core expertise. For an iOS engineer like myself, they excel at web development tasks I'd otherwise need to research extensively. The tools serve as force multipliers for exploring adjacent technologies rather than replacing deep domain knowledge (I'm convinced this will change super quickly though).

The interaction paradigm represents the more fundamental shift. We're moving toward intent-driven development where you describe desired outcomes rather than specifying implementation details. This abstraction works well for certain problem domains while remaining limited in others.

For now, AI development tools are powerful assistants rather than replacements, with effectiveness varying significantly by technology stack and problem complexity. It is starting to feel like you can start a company with less employees, particularly around building a website, blog, doing marketing, etc. The non-coding aspect is something I will explore more of next (I just connected Claude to my gmail inbox...).

**Disclaimer:** This post was written by me but proofread and formatted by Claude. Yet another way Claude helps me lately, especially as a non-native English speaker!


Jernej Štrasner

Blog

Building an LLM Gateway: Lessons from Tool Calling Across Models

Learning How Chunking, Embeddings, and Vector Databases Actually Work

AI Development Tools: A Pragmatic Assessment