Chunking and vector search

Asha’s handbook for Chai & Chapters has grown to forty pages: opening hours, pets, refunds, Wi-Fi, parking, book club, allergies. A customer types, “Can I bring my dog on Sunday?” The answer is in there somewhere. But where? And how much of the handbook should the app hand to the model?

In the last lesson, search “found the right chunks” as if by magic. Today we open that box. It comes down to two decisions: how you cut the book, and how you measure which piece is closest to the question.

Why not paste all forty pages into every prompt? Three reasons. It may not fit in the context window. Even if it fits, every question pays for thousands of extra tokens. And models often pay less attention to details buried in the middle of a very long prompt.

So we cut the handbook into small pieces, find the few that matter for this question, and send only those.

In the embeddings lesson, you saw that a list of numbers can place text on a map of meaning. “Dog” sits near “puppy” and “pet”, and far from “Wi-Fi”. The same trick works for whole paragraphs and whole questions.

So each chunk goes through an embedding model once, and gets its own point on the map. Those points, with the original text attached, go into a special store. When a question arrives, it goes through the same embedding model and becomes a point too.

Now search becomes a question of distance: which chunk points sit nearest the question’s point? The database measures closeness, often with a score called cosine similarity, and returns the best few. A higher score means closer meaning.

Here is the lovely part. The customer typed “dog”. The handbook says “pets”. A keyword search looking for the letters d-o-g finds nothing. Vector search knows that dog and pet live in the same neighbourhood of the map, so the pets section comes up first.

With millions of chunks, comparing the question with every single point would be slow. So vector databases use clever shortcuts that find almost-nearest points very fast, called approximate nearest neighbour search. The idea is enough for now: a tiny bit of accuracy is traded for a lot of speed.

Now the decision that quietly makes or breaks RAG: how big should each chunk be? Picture the pets part of the handbook. One line says well-behaved dogs are welcome. A nearby line adds a special rule for Sundays.

Cut too small, and those two lines land in different chunks. Search finds “dogs are welcome”, misses the Sunday rule, and the helper happily says yes. Cut too big, and the Sunday rule is in there, but buried among Wi-Fi details and parking, inside a long and costly prompt.

Medium chunks win here because each one holds a whole idea: the pets rule and its Sunday exception, together, found with an 86% match. Huge chunks match only weakly, around 44%. That is the real goal. Not a magic number, but chunks that each make sense when read on their own.

1. 1

   Cut along the document’s own shape

   Split by headings, sections and paragraphs first. A section on pets should stay one chunk if it fits.

2. 2

   Start at a few hundred tokens

   Many teams begin somewhere around 200 to 800 tokens per chunk, then adjust. Short FAQs need less; dense legal text may need more.

3. 3

   Add a little overlap

   Share roughly 10 to 20 percent between neighbouring chunks, so nothing important falls into the gap.

4. 4

   Keep labels on every chunk

   Store the title, page and date with each chunk. They help search, and let the answer cite its source.

5. 5

   Test with real questions

   Write twenty real customer questions and check that the right chunk lands in the top few. Change one thing at a time.