
I’ve been building products and tools for years — mostly iOS, lots of scripting, some backend, a lot of “make it work and ship it.” Recently I started poking into the AI space. Not just “call an LLM and magic happens,” but the *internals*. I want to understand enough to eventually build something serious on top of this tech.

One of the first things I kept hearing about in RAG (retrieval-augmented generation) setups was **chunking** and **embeddings** — and how they all tie into a **vector database**. I figured I’d document my beginner mental model.

## Step 1: Chunking
This part made sense quickly. You have long documents, but your embedding model (and later the LLM) has token limits. So you split the text into smaller **chunks**.

- Typical chunk size: ~500–800 tokens (2,000–3,200 characters in English)  
- Overlap: 10–20% so context flows between chunks  
- Too big = you get noisy, unfocused vectors. Too small = you lose context.

I now think of chunking as just a preprocessing step to keep everything bite-sized for the next phase.

## Step 2: Embeddings
Once you have chunks, you pass each one through an **embedding model**. This is where my “OK, but how?” curiosity kicked in.

The model outputs a **vector** — literally just an array of floats, like: [0.0213, -0.1342, 0.0876, …]

The magic is that similar meanings end up with vectors that are close together in this high-dimensional space. “Car” and “Automobile” will be neighbors. “Car” and “Banana” will be far apart.

It’s nothing like Levenshtein distance (string edits). Instead, embeddings are trained so semantic closeness translates to geometric closeness.

## Step 3: Storing in a Vector DB
Each chunk’s vector goes into a **row** in the vector database, alongside the original text and metadata. The DB builds an index so later we can search fast.

The vector DB doesn’t care about tokens or text anymore — it’s just arrays of numbers and math.

## Step 4: Retrieval
Here’s the clever bit:

1. User asks a question.  
2. You embed the question into its own vector (same model).  
3. The vector DB finds the **nearest neighbors** to that query vector.  
4. You fetch the corresponding chunks and feed them into your LLM for the final answer.

The naive way is to just scan all rows and pick the highest cosine similarity score (cosine similarity = dot product normalized for vector length). That works for small data, but with millions of vectors it’s too slow.

## Step 5: How Lookup is Actually Fast
Most vector DBs use an **approximate nearest neighbor** index — HNSW is the popular one. My mental model:

- Imagine all your points as cities on a map.  
- The DB builds a set of “highway” layers to jump across the space quickly, then “street” layers to refine locally.  
- When searching, it hops across highways, drops down to streets, and zeroes in on the closest points without visiting every “city.”

This gets you *almost* the same result as scanning everything, but way faster.

## Dot Product & Cosine Similarity (The Core Math)
The dot product is just multiplying each vector component, summing it up.  
Cosine similarity is the dot product divided by the product of vector lengths — which basically strips out magnitude and keeps only the **angle** between the vectors.

For embeddings, angle matters more than length, because we care about **direction in semantic space**, not absolute size.

## Final Mental Picture
- Chunk → embed → store in vector DB → embed query → find nearest vectors → send chunks to LLM.  
- Embeddings are just numbers.  
- Vector search is just “find points closest to this one” with clever shortcuts.  
- The rest is plumbing and product thinking.

I’m only scratching the surface, but already this feels a lot less like magic and more like a set of tools I can reason about.

Jernej Štrasner

Learning How Chunking, Embeddings, and Vector Databases Actually Work