RAG Chunking Strategies: How to Split Documents for Retrieval

The search step finds pieces that were cut in the wrong place, and the model answers from half a fact. Good RAG chunking strategies prevent this by cutting where the document itself changes topic.

Picture a common setup. A farmer helpline in Maharashtra answers crop questions that farmers send by phone call or text message. Behind it sit about 35 crop-care manuals and 60 government scheme PDFs, roughly 9,000 pages. The setup is made up, but the problem is real.

The helpline's RAG bot gets a question from a cotton farmer. Whiteflies are on his crop, and he wants to know how much of a spray to mix per litre of water. The manual's dose table says 2 ml per litre for whitefly. The bot answered 3 ml, the dose from the aphid row just below.

A dose 50 percent too strong wastes spray and can harm the crop. The cause was the cutting rule. The first version was the naive one: it cut every document every 2,000 characters, wherever that landed.

One cut fell right after the word whitefly in the dose table. The search found the next piece, full of dose and spray words, which began with an unlabelled 2 ml and then the aphid row's 3 ml. The model saw no whitefly label and picked the labelled dose.

The team then collected 40 real farmer questions, each with its correct answer. The right text reached the bot's top 4 search results for only 22 of them. This guide follows the team as it raises that number, one chunking idea at a time.

Six words carry this whole guide. Each one gets a plain meaning and an example from the helpline. Our guide RAG Pipelines Explained covers the full flow around them; this guide covers only how you cut the documents.

Chunk. A chunk is one piece of a document that is stored and searched on its own. Example: the whitefly row of the dose table, with the table's header row, could be one chunk.

Token. A token is a whole word or a piece of a word, the unit a model reads and counts. Chunk sizes and model input limits are usually counted in tokens. How many words fit in 100 tokens depends on the tokenizer, the tool that does the cutting into tokens.

Language changes the count too. A 2023 study found the same text can need very different numbers of tokens in different languages, up to 15 times more in some cases (Petrov et al., 2023). So measure Hindi or Marathi chunks with your own model's tokenizer.

Embedding. An embedding is a list of numbers that stands for a text's meaning. Texts with similar meanings get similar numbers. That lets search match a question to chunks by meaning, not only by shared words.

Retrieval and top-k. Retrieval is the search step that finds chunks for a question. Top-k means the k best matches it returns. Example: with k set to 4, the whitefly question brings back 4 chunks.

Overlap. Overlap is text that two neighbouring chunks share. Example: the last 50 tokens of chunk 41 are also the first 50 tokens of chunk 42. A short sentence cut at that border still appears whole in one of them.

Metadata. Metadata is extra information stored beside a chunk, apart from its text. Example: file cotton-manual.pdf, page 112, section Pests / Whitefly.

RAG chunking strategies are the rules that decide where one chunk ends and the next begins. This guide covers five, from the least work to the most:

• Fixed-size: cut every N tokens, usually with a little overlap.
• Recursive: cut at the biggest natural break that makes pieces small enough, trying a blank line before a sentence end.
• Structure-aware: cut along the document's own headings, lists and tables.
• Semantic: cut where the meaning of neighbouring sentences changes, found by comparing their embeddings.
• Parent-child: search small chunks, then send the model the bigger section each one came from.

They are layers more than rivals. In the helpline's final setup, structure-aware cuts come first, recursive cuts handle sections that are still too long, and parent-child sits on top. First, though, you need to see why size matters at all.

Search can only return whole chunks. Too small, and a chunk loses the facts around it. Too big, and the answer is buried among unrelated text.

Think of cutting clippings from a newspaper to keep in a file. Cut out one line and you lose its headline, so you no longer know what it is about. Keep the whole page, and the crop prices you wanted sit beside cricket scores and film ads.

The parts map like this. A clipping is a chunk, the file is your search index (the stored, searchable set of chunks), and the headline is the context a chunk needs. Where the picture breaks: you read a clipping yourself, but search sees only its embedding.

Now take the whitefly question. A tiny 40-token chunk might say only "Mix 2 ml per litre, spray in the evening". It never says which pest or which crop, so the model has to guess.

A huge 3,000-token chunk might hold the whole pest chapter, with 12 pests in it. Its one embedding has to stand for all 12 topics, so it may match the whitefly question less closely than a focused chunk. Even when it is found, the model must pick one row out of hundreds of lines. If it is longer than the embedding model's maximum input, the rest may be cut off before embedding (see the library defaults section).

Size also changes your bill. If your model service charges per input token, 4 chunks of 3,000 tokens add ten times as many billed tokens as 4 chunks of 300.

Long inputs bring a second problem. A 2023 study tested question answering over many documents at once. Models often did best when the useful text sat at the start or end of the input, not the middle (Liu et al., 2023).

The diagram follows one question to three chunk sizes. Notice that only the middle size hands the model both the dose and the pest it belongs to.

The right size is not one magic number. It depends on how your documents are written and how people ask. Start with a few hundred tokens for manual-style text, then let a test on your own questions decide.

Fixed-size chunking cuts text every N tokens, no matter what the text says. Overlap repeats the last few tokens of each chunk at the start of the next, so a sentence no longer than the overlap survives whole in one of them.

The team's first fix was to count in tokens instead of characters, with 300-token chunks and 50 tokens of overlap. Each new chunk then starts 250 tokens after the last one. A 1,000-token page becomes 4 chunks, starting at tokens 0, 250, 500 and 750, and the last one holds 250 tokens.

Overlap has a price. You store about 20 percent more chunks, because each chunk moves forward 250 tokens instead of 300, and 300 divided by 250 is 1.2. Overlap is a safety net for bad cuts. It does not stop the bad cuts.

On the team's test set, this raised the hits from 22 to 27 out of 40 (made up). The dose table was still cut in half, though. Fixed-size chunking cannot see tables, headings or anything else about the text.

Recursive chunking tries the biggest natural break first, such as a blank line, then smaller ones, until every piece fits your size limit. Sentence splitting and paragraph splitting each use one kind of break, so one over-long sentence or paragraph stays over the limit.

Sentence splitting cuts at full stops and question marks, and for Hindi and Marathi at the danda (।), the vertical line that ends a sentence. Then it packs whole sentences into each chunk until the next one would not fit. Short forms like "approx." can fool a simple splitter into cutting too early.

A splitter that knows only the full stop fails Hindi the other way. It never cuts at the danda, so a whole Hindi paragraph comes out as one giant sentence. Try any splitter on a two-line Hindi note before you trust it.

Paragraph splitting cuts at blank lines. It keeps related sentences together, which is good. The problem is size: one paragraph in the cotton manual has 20 words, and another has 900.

Recursive chunking fixes that. Recursive means a step that repeats itself on smaller and smaller pieces. Follow one oversized section through the loop in the diagram.

The 1,400-token section first splits at blank lines into paragraphs. One paragraph is still 600 tokens, so it splits again at line breaks, then at sentence ends. Almost nothing is ever cut at a space. Afterwards, small neighbouring pieces are joined back up to the limit, so you do not get 15-token chunks.

At least one widely used open-source library ships a recursive splitter, and its default break order differs from ours. In its release current on 19 October 2026, it tries a blank line, a line break, a space, then single characters, with no sentence step. In our test, a long paragraph with no line breaks was cut at spaces, in the middle of sentences. On the team's test set, recursive chunking at 300 tokens with 50 overlap reached 30 out of 40 (made up).

Turn the PDF into elements first, such as headings, paragraphs, lists and tables, and then cut along those elements. That way a chunk never starts in the middle of a table or mixes two sections.

PDF was built to keep how a page looks, so a plain PDF does not necessarily record the document's structure. It has no explicit heading marker, so a big bold line only looks like a heading.

Tagged PDFs add a tree of structure marks for parts such as paragraphs, but you cannot count on every file having them. Plain text extraction can lose table structure too. We tested the command-line tool pdftotext (version 24.02) on a small dose table. In its default mode it put each cell on its own line, with blank lines between, so only the line order tied a pest to its dose.

So the team used a PDF parser that detects layout: it labels each block as a title, paragraph, list or table. Some older scheme PDFs were scanned photos of paper. Those first need OCR (optical character recognition), which turns a picture of text into real text.

With the elements in hand, the team followed these rules:

• Start a new chunk at every heading, so two sections never share a chunk.
• Keep each table whole if it fits the size limit. If it does not, split it by rows and repeat the header row at the top of every piece.
• Keep a numbered list, such as the documents needed to apply for a scheme, together in one chunk.
• Put the heading path at the top of every chunk's text, for example "Cotton manual / Pests / Whitefly".
• Use recursive chunking only inside a section that is still too long.

The heading path rule matters more than it looks. The whitefly row on its own says "Whitefly | 2 ml per litre", and never says cotton. With the heading path on top, the chunk now holds the words a farmer actually uses.

A 2024 study on financial reports did something similar. It started a new chunk at every title or table and kept each table whole. On 141 questions, answers judged by hand were right up to 53 percent of the time. Its best fixed-size baseline, 512-token chunks, reached 48 percent (Jimeno Yepes et al., 2024).

Read that result with care. The study's element chunks also carried extra keywords, summaries or a prefix (a copy of the chunk's first two sentences). Each of those three versions found the right page less often than every fixed-size baseline. Farm manuals are not financial reports, so treat it as a reason to test, not a promise.

For the helpline, structure-aware chunking reached 34 out of 40 (made up), and the whitefly question finally got 2 ml per litre.

Store whatever helps you filter, cite and debug: at least the source file, page, heading path and document date. Then add the fields your users' questions depend on. Here is what each helpline chunk carries:

• Source file and page, such as cotton-manual.pdf, page 112, so the bot can say where an answer came from.
• Heading path, such as Pests / Whitefly, so you can see where the chunk sits in the document.
• Document type, manual or scheme, so a question about subsidy forms searches only scheme PDFs.
• Crop, such as cotton, soybean or orange, so a cotton question does not get a soybean answer.
• Year or version, because the 2024 scheme PDF had a different last date from the 2026 one.
• Language, because some PDFs are in Marathi and some in English.
• Chunk id and parent id, which parent-child chunking uses later in this guide.

Check what your setup actually embeds. If it embeds only the chunk's text, a crop name stored only as metadata cannot help the search match. That is why the heading path goes in both places.

Not reliably. Semantic chunking cuts where the meaning of neighbouring sentences changes, and sometimes that helps. But a 2024 study concluded its extra computing cost was not justified by consistent gains over fixed-size chunks (Qu et al., 2024).

Here is the version the study calls breakpoint-based. The program splits text into sentences and makes an embedding for each one. It measures how far each sentence's meaning is from the next. Where that distance passes a threshold, a set cut-off, it starts a new chunk.

Example: one scheme PDF has a long paragraph that moves from who can apply to how much the premium is, with no heading between them. Semantic chunking can spot that change of topic and cut there.

The costs are real. You need one embedding per sentence while building, instead of one per chunk. You must tune the threshold. Chunk sizes can also come out very uneven.

The study's results were mixed, not one-sided. In its document-finding test, semantic chunkers won on documents the authors built by stitching unrelated short texts together, so the topic jumped often. On the original documents, fixed-size chunks of consecutive sentences won. In its evidence-finding and answer tests, the differences were small.

Pause and predict before reading on. Semantic chunking reads meaning, and the manuals are full of meaning. Guess whether it beats structure-aware chunking's 34 out of 40.

On the helpline's test set, semantic chunking reached 31 out of 40, below structure-aware chunking at 34, and its build took six times longer (both made up). The manuals have good headings, and headings already mark where the meaning changes.

Parent-child chunking searches small chunks for a sharp match, then hands the model the larger parent section the match came from. You get precise search and full context at the same time.

Back to the clippings. You file each small clipping with a note of which page it came from. You search the small clippings because they are easy to match, then pull out the full page to read the story around it.

For the helpline, child chunks are about 100 tokens: one table row with its header, or one short paragraph. The parent is the whole section, about 800 tokens. Each child stores its parent's id in its metadata.

Now the farmer gets the dose plus the manual's advice around it: spray in the evening, and wait seven days before spraying again. That advice sits in the paragraph under the table, which the child chunk alone would miss.

Two details matter. First, if three of the top four children share one parent, send that parent once. Second, keep parents a sensible size and send only a few, or you are back to the huge-chunk problem.

The search half has research behind it. A 2023 study found that indexing small, self-contained facts retrieved better than indexing whole passages (Chen et al., 2023). Its small units were rewritten by a model to stand on their own, which plain small chunks are not. It also did not test the parent step, so test it yourself.

With parent-child chunking, the helpline reached 37 out of 40 (made up). The cost is storing two sizes of chunk and sending more tokens per question.

Build a small test set of real questions with known answers. Run each strategy on the same documents, and count how often the right text reaches the top few results. The best score at a cost you can afford wins.

The helpline's agronomist, its crop expert, built the test set in one afternoon:

1. 1

   Pick real questions

   Take 40 questions from last month's call log, not invented ones, because farmers word things in ways engineers do not expect.

2. 2

   Cover the hard cases

   Include table questions, scheme date questions, Marathi questions and two questions whose answer is in no document.

3. 3

   Write the answer text

   For each question, copy a short phrase that must appear in a good chunk, such as "2 ml per litre", plus the file and page.

4. 4

   Freeze the set

   Keep the same questions for every strategy, so the scores can be compared fairly.

5. 5

   Run and record

   Run each strategy, record the score and the number of chunks, and save the misses for the expert to read.

The score is called hit rate at k. A 2020 retrieval paper calls it top-k retrieval accuracy: the share of questions whose top k results contain text that answers them (Karpukhin et al., 2020).

import re

def hit_rate(test_set, search, k=4):
    hits = 0
    for item in test_set:
        texts = [t.lower() for t in search(item["question"], k)]
        if any(item["must_contain"].lower() in t for t in texts):
            hits += 1
    return hits / len(test_set)

# Check it with three chunks and a tiny word-overlap search
chunks = ["Cotton / Pests / Whitefly: mix 2 ml per litre, spray in the evening",
          "Cotton / Pests / Aphids: mix 3 ml per litre",
          "Crop insurance scheme: apply by the last date in the notice"]

def search(question, k):
    words = set(re.findall(r"\w+", question.lower()))
    score = lambda c: len(words & set(re.findall(r"\w+", c.lower())))
    return sorted(chunks, key=score, reverse=True)[:k]

tests = [{"question": "whitefly on cotton, how many ml per litre",
          "must_contain": "2 ml per litre"},
         {"question": "last date for crop insurance",
          "must_contain": "31 july"}]   # no chunk holds this date: a miss
print(hit_rate(tests, search, k=1))

What just happened: the first question's top chunk contains "2 ml per litre", so it counts as a hit. No chunk holds the insurance date, so the second question is a miss. One hit in two questions prints 0.5.

Notice the check uses answer text, not a chunk id. Every chunking change creates new chunks with new ids. A text check keeps working across every strategy you try.

The diagram shows the loop. The questions and documents never change between runs. Only the chunking strategy does.

Here are the helpline's results. Every number is made up, and your documents will give different numbers. That is the whole reason to run the test.

Hit rate checks only the search step. A hit does not promise a good answer, so the expert also read 15 full bot answers for each of the top two strategies. Our RAG evaluation metrics guide covers that kind of answer checking.

Both fit in a few lines of plain Python. In these demos, words stand in for tokens, so the code runs with nothing installed. In real use, count tokens with your embedding model's tokenizer.

def fixed_size_chunks(tokens, size=300, overlap=50):
    step = size - overlap                      # move forward 250 tokens each time
    last_start = max(len(tokens) - overlap, 1)  # stop before a chunk of pure overlap
    return [tokens[start:start + size] for start in range(0, last_start, step)]

# Check it: words stand in for tokens here
page = [f"w{i}" for i in range(1000)]          # a 1,000-token page
chunks = fixed_size_chunks(page)
print([c[0] for c in chunks])                  # where each chunk starts
print([len(c) for c in chunks])                # tokens in each chunk

Run it and you should see the starts w0, w250, w500 and w750, with sizes 300, 300, 300 and 250. The range stops 50 tokens early, so you never get a last chunk made only of overlap.

The recursive version below tries each break in order and only moves to a smaller break when a piece is still too big. Then it joins small neighbours back up to the limit.

BREAKS = ["\n\n", "\n", ". ", "। ", " "]       # blank line, line break, sentence ends, space
# simple demo: it skips question marks, and "approx. " fools it

def fits(text, limit):
    return not text.split()[limit:]           # no words left after the first `limit`

def split_keep(text, sep):
    parts = text.split(sep)                   # cut at sep, but keep sep on each piece
    return [p + sep for p in parts[:-1]] + [parts[-1]]

def recursive_split(text, limit, breaks=BREAKS):
    if fits(text, limit) or not breaks:
        return [text]
    pieces = []
    for part in split_keep(text, breaks[0]):
        if part:
            pieces += recursive_split(part, limit, breaks[1:])
    return pieces

def merge_small(pieces, limit):
    chunks, current = [], ""
    for p in pieces:
        if current and not fits(current + p, limit):
            chunks.append(current)
            current = ""
        current += p
    return chunks + [current] if current else chunks

# Check it on a made-up section: words stand in for tokens again
sentence = " ".join(["word"] * 14) + " end. "            # a 15-word sentence
section = (" ".join(["intro"] * 40) + "\n\n"             # 40-word paragraph
           + sentence * 8 + "\n\n"                       # 120-word paragraph
           + " ".join(["tail"] * 57))                    # 57-word paragraph
pieces = recursive_split(section, limit=100)
chunks = merge_small(pieces, limit=100)
print(len(chunks), "chunks, sizes", [len(c.split()) for c in chunks])
print("text kept exactly:", "".join(chunks) == section)

Expected output: 3 chunks of 100, 60 and 57 words, then "text kept exactly: True". The 120-word paragraph was too big, so it was cut at sentence ends, never mid-sentence.

What just happened: the merge step glued the 40-word first paragraph to the first four sentences of the second. If you want chunks never to cross a paragraph border, merge only within one paragraph, and test both versions.

Defaults differ a lot between libraries, so read yours before trusting a chunk size. We read the source of three widely used open-source libraries in their releases current on 19 October 2026, and ran one of them.

• One library's recursive splitter counts size in characters by default, not tokens: 4,000 characters, with 200 characters of overlap.
• Another library's sentence splitter counts in tokens by default: 1,024 tokens, with 200 tokens of overlap. Those tokens come from its own built-in tokenizer, which may not match your embedding model's.

So a setting of 300 can mean 300 characters in one tool and 300 tokens in another. Defaults like these can change between releases, so pin your version.

The overlap setting is a target, not a promise

In the character-counting splitter, overlap is built from whole pieces carried over from the previous chunk. If the cut falls between paragraphs that are each bigger than the overlap, nothing is carried over.

We ran it with a 300-character limit, 50 characters of overlap and six paragraphs of 202 characters each. It made six chunks, and neighbours shared 0 characters. On one long line of short words, neighbours shared 49.

Its documentation page for this splitter calls overlap a target and does not mention the zero case. Treat that as undocumented as of 19 October 2026, and pin your version.

Long chunks can be cut short before embedding

Embedding models have a maximum input length, counted in tokens. In one widely used open-source embedding library, longer text is cut off by default, with no error. We read this in its releases current in October 2026.

Anything past the limit never reaches the embedding. A big chunk can then be found only by its first part, while the rest sits in the index unseen.

1. 1

   Find the size unit

   Look up your splitter's length function, the code that measures each piece. If it counts characters, convert your token target before setting the size.

2. 2

   Measure real overlap

   Split a few real documents and count the text shared by each pair of neighbours. Expect some pairs to share nothing.

3. 3

   Compare with the model limit

   Count the tokens in your longest chunk with the embedding model's own tokenizer. It should be below the model's maximum input length.

A strong answer starts from the documents and the questions, picks a simple default, and proves every upgrade with a test. Interviewers want to see that you know the trade-offs, not a list of method names.

1. 1

   Ask about the documents

   Find out if they have headings, tables, scans or several languages, and how users word their questions.

2. 2

   Start with a simple default

   Propose recursive chunking at a few hundred tokens with small overlap, cutting along headings and tables where they exist.

3. 3

   Add metadata

   Store the source, page, heading path, date and filter fields, so the system can filter results and cite its answers.

4. 4

   Upgrade only where needed

   Add parent-child chunking if answers need surrounding context. Mention semantic chunking only for unstructured text.

5. 5

   Prove it with a test set

   Describe a frozen set of real questions, hit rate at k for each strategy, and a re-run whenever documents or chunking change.