RAPTOR: Hierarchical Summaries for Long Documents

Plain RAG often misses parts of the answer to a broad question, because it searches small pieces of text and a broad answer is spread across many chapters. The search returns a few similar passages from one corner and misses the rest. RAPTOR RAG, the method this guide explains, adds a tree of summaries so search can see whole chapters.

RAG is short for retrieval-augmented generation. A search step finds passages related to your question and pastes them into the prompt, the text sent to the model. The model, a large language model that writes text, then answers from those passages.

Picture a common setup. A small team at a consulting firm reviews detailed project reports, or DPRs, for state road and bridge projects. The setup is made up, but the problem is real.

A DPR runs to about 300 pages, with chapters on traffic, land, soil, design, cost, environment and risks. The team keeps 40 of them in an archive and has built a question-answering tool on top using plain RAG.

Small questions go well. An analyst asks for the design speed on a coastal highway bypass near Puri, and the tool returns the right page with the right number.

Then she asks the question the client actually cares about. What are the main risks to this project, and how does the report plan to handle them? Here is what happens.

• The tool returns five passages, all from the land acquisition chapter, because they use the word risk most often.
• It misses the flood risk, which sits in the hydrology chapter and never uses the word risk.
• It misses the cost overrun risk, which is spread across three tables in the finance chapter.
• The answer sounds confident and complete, yet it covers about one third of what the report says.

Reading all 300 pages by hand takes the analyst two days per report. The team needs search that can see the whole report at different zoom levels. That is the problem RAPTOR was built for.

You need five words: chunk, token, embedding, cluster and node. Each gets a plain meaning and an example from the DPR archive.

Chunk. A chunk is a small piece of a document, cut so it can be searched on its own. Example: one paragraph about the 2019 flood level near the Puri bypass.

Token. A token is the unit a model reads, often a whole word or part of one. Model limits, prices and the chunk sizes in this guide are all counted in tokens.

Embedding. An embedding is a list of numbers that captures what a piece of text means. Texts about similar things get similar numbers. Example: a chunk about drainage culverts sits close to a chunk about flood levels.

Cluster. A cluster is a group of items that sit close to each other. Example: twelve chunks about water, rain and culverts form one cluster, even though they sit on different pages.

Node. A node is one entry in the tree. It is either an original chunk or a summary written by the model. Each node keeps its text and its embedding, and a summary node also keeps links to its children, the nodes it summarises.

RAPTOR RAG indexes a long document as a tree: the bottom level holds the original chunks, and each level above holds summaries of groups below. Search can then pick the level of detail each question needs.

The name stands for Recursive Abstractive Processing for Tree-Organized Retrieval. It comes from a January 2024 paper by Sarthi and colleagues at Stanford, published at the ICLR 2024 conference. Recursive means the same step repeats on its own output. Abstractive means the summaries are written in new words, not copied sentences.

Think of a week-long Bhagwat katha at a temple. Each evening the kathavachak, the storyteller, tells several episodes in full detail. Before a new evening he recaps earlier days in a few minutes, and on the last day he retells the whole week in a few sentences.

Where the picture breaks: the storyteller recaps in story order. RAPTOR groups by meaning, so one summary can join passages from page 88 and page 142. One chunk can also sit under two different summaries.

Here is part of the tree for the coastal highway DPR. Read it from the bottom up. Each arrow means the lower node is summarised into the upper one.

The flooding summary joins a hydrology chunk and a design chunk, which plain chunking would keep apart. The groups follow meaning, not page order. That is why the tree can answer the risk question.

RAPTOR builds the tree bottom up: it embeds the chunks, clusters similar ones, summarises each cluster, then repeats those steps on the summaries. It stops when the top level is too small to cluster further.

1. 1

   Cut the document into chunks

   The paper uses chunks of 100 tokens. A sentence that would cross the limit moves whole to the next chunk, so no sentence is cut in half.

2. 2

   Embed every chunk

   An embedding model turns each chunk into a list of numbers. Our made-up 300-page DPR gives about 2,000 chunk embeddings.

3. 3

   Group similar chunks

   Clustering puts chunks with similar meaning together, wherever they sit in the report. The method allows one chunk to join more than one group.

4. 4

   Summarise each group

   A model writes a short summary of each group. Each summary becomes a new node, with links to the chunks it covers.

5. 5

   Repeat on the summaries

   The new summaries are embedded, grouped and summarised again. Each round makes a smaller, higher level.

6. 6

   Stop near the top

   The paper repeats until further clustering is not feasible. Every node from every level is then saved in a vector store.

A vector store is a database that holds embeddings and finds the ones closest to a question. After the build it holds chunks and summaries side by side.

Summaries are much shorter than what they cover. Across the paper's datasets a summary averaged 131 tokens, about 0.28 of the combined length of its children. A parent node had 6.7 children on average.

Those averages let you estimate the tree. With 6.7 children per summary, 2,000 chunks give about 300 level 1 summaries, then 45, then 7. That is roughly 350 summaries per report, our own rough estimate.

How the grouping works

The grouping step uses two well-known methods. UMAP, a dimension reduction method, squeezes each long embedding into a few numbers. The paper does this because distance measures can behave poorly on very long lists of numbers.

A Gaussian Mixture Model, or GMM, then finds the groups. A GMM does soft clustering: instead of one group per chunk, it gives each chunk a probability for every group. A chunk joins every group where that probability passes a threshold, 0.1 in the reference code.

Soft clustering suits DPRs. A paragraph on raising the road embankment talks about flooding and about cost. Soft clustering allows it to sit under both summaries.

How many groups should there be? RAPTOR tries several counts and scores each with BIC, the Bayesian information criterion. BIC rewards a good fit but punishes extra groups, and the lowest score wins.

One more guard keeps summaries possible. If a group's text is too long for the summarising model, RAPTOR clusters that group again into smaller groups. The reference code sets that limit at 3,500 tokens.

RAPTOR's preferred search is collapsed tree: put every node from every level in one pool and take the closest ones until a token budget is full. The paper also tests a top-down walk called tree traversal.

Tree traversal starts at the top level. It keeps the few summaries closest to the question, then looks only at their children, level by level, down to the chunks. Because it keeps the same number of nodes per level, the mix of summary and detail is the same for every question.

Collapsed tree ignores the levels during search. Summaries and chunks compete in one ranked list, scored by cosine similarity, a measure of how closely two embeddings point the same way. A broad question tends to pull in more summaries, and a narrow one mostly chunks.

Back at the katha, a listener asks two questions. For what the whole story teaches, the storyteller answers from his last-day retelling. For one scene, he goes straight to that episode without climbing down from the top.

The paper compared both methods on 20 documents from the QASPER question set, and collapsed tree did better consistently. Its main experiments used collapsed tree with a 2,000-token budget, roughly the top 20 nodes.

Collapsed tree has a cost: it scores the question against every node in the tree. The paper notes that fast nearest-neighbour libraries such as FAISS make this efficient.

In the original paper, adding the tree helped on all three long-document question sets tested. The authors credit the upper summary levels for questions about themes or needing several steps of reasoning. Your documents may behave differently, so test on your own questions.

Each test pairs a retriever, the search step, with a reader model that writes the answer. The paper reports these results, as of its January 2024 version.

• QuALITY has multiple-choice questions on passages of about 5,000 tokens. With the strongest reader model tried, RAPTOR reached 82.6% accuracy against 62.3% for the previous best published system.
• On QuALITY's hard subset, the same setup scored 76.2% against 54.7% for that earlier system.
• QASPER asks questions about full research papers. With the same strong reader, RAPTOR scored 55.7 F1, a measure of word overlap with the right answer, against the previous best of 53.9.
• NarrativeQA asks about whole books and film scripts. RAPTOR with a smaller open reader model set a new best METEOR score, another answer-matching measure.
• In controlled tests, adding the tree improved each of three retrievers, including BM25 keyword search, on all three datasets.

One finding matters more than the scores. Between 18.5% and 57% of the nodes that collapsed tree retrieved came from summary levels, depending on the dataset and retriever. Summaries were really used, not just stored.

The paper also shows a fairy tale example. The question was how Cinderella finds a happy ending. A dense retriever, a search model that compares embeddings, returned passages about her first transformation, such as her old shoes becoming glass slippers. RAPTOR's context included the prince's search for the slipper's owner.

Why not paste all 300 pages into the prompt instead? Even when a model's context window, the amount of text it can read at once, is big enough, you pay and wait for every token sent. A 2023 study, Lost in the Middle, also found the models it tested often did worse on facts in the middle of long inputs.

Here is how the three choices compare for the team's archive of 40 DPRs. The costs and times are made up to show the shape of the trade-off, not measured.

RAPTOR costs a little more per question because the context now holds summaries too. The real price is paid once, at build time.

RAPTOR costs one model call per summary at build time, and part of the tree goes stale whenever a document changes. The paper reports that build time and tokens grow in a straight line with document length.

That straight line is good news. Doubling the report roughly doubles the cost, rather than making it explode. Our estimate of 350 summaries per DPR means about 14,000 summary calls for 40 DPRs, paid once per version.

In tokens, each level 1 call reads about 6.7 chunks and writes about 131 tokens. Adding up all levels gives roughly 2.6 lakh (260,000) tokens of model input and output per report, again our own rough estimate.

Updates are the harder part. DPRs get revised, and the state department sends a corrigendum, a short official correction, every few months. A changed chunk makes every summary above it stale.

• Rebuild the whole tree for that one report. This is simple and safe, and fine when updates are rare.
• Rebuild only the path above the changed chunks. This is cheaper, but the old groups may no longer fit the new text.
• Keep the old tree until a nightly rebuild, and tag answers from changed reports as possibly outdated.

The team picks the first option. Each report gets its own tree, so a corrigendum to one DPR rebuilds about 350 summaries, not the whole archive.

Store the source version next to every node, such as the DPR revision number. Then the system can refuse to mix nodes from an old tree with chunks from a new one.

The main risks are summaries that state things the source never said, summaries that drop the one detail you needed, and trees that go stale. Each needs a check in your pipeline.

A hallucination is when the model states something false as if it were true. The authors sampled 150 summary nodes across 40 stories and found 6, or 4%, with minor hallucinations. These did not spread to parent nodes and had no visible effect on their question-answering results.

Four percent still matters for a DPR. A summary that says the bridge is designed for a 100-year flood, when the report says 50, could mislead a client. So the team adds guards.

• Numbers come from chunks. If an answer quotes a figure, the system checks that the figure appears in a retrieved chunk, not only in a summary.
• Answers name their pages. Each summary keeps links to its chunks, so the answer can cite the original page numbers.
• People spot-check. Every week an analyst reads 20 random summaries against their chunks and logs any mistake.
• Tables get care. Cost tables lose meaning when squeezed into prose, so they stay as chunks with a short plain description.

Bad input makes a bad tree. If the PDF parser reads two columns as one, the chunks are garbled and every summary above them is too. Fix parsing before building anything on top.

Use RAPTOR when documents are long, change rarely, and people ask broad questions that cross many sections. For short documents or pure fact lookups, plain chunk RAG is cheaper and simpler.

The team checks its real questions before deciding. It labels 200 saved questions from the analysts. About 60% are fact lookups, and 40% ask about themes, risks or comparisons across chapters.

Signs that RAPTOR will help are easy to spot. Look for these four.

• Documents run to hundreds of pages, like DPRs, contracts, policy manuals or books.
• People ask what, why and how questions about the whole document, not only for one number.
• The documents change rarely, so the build cost is paid once per version.
• Plain RAG answers sound complete but miss whole sections when you check them.

Signs it will not help: short documents, a stream of fast-changing records like tickets or chats, or questions that are almost all exact lookups. In those cases, better chunking, reranking or query rewriting usually help more.

You can run both core steps on a laptop. We ran these samples on Python 3.13 with umap-learn 0.5.12, scikit-learn 1.9.1 and NumPy 2.5.3 on 19 October 2026.

The grouping step

This is a simplified, one-stage version of RAPTOR's grouping. The reference code first finds broad groups, then clusters again inside each one. The test data is fake: three made-up topics with 40 embeddings each.

import numpy as np
import umap
from sklearn.mixture import GaussianMixture

def group_nodes(vectors, threshold=0.1, max_groups=50):
    small = umap.UMAP(n_neighbors=10, n_components=10, metric="cosine",
                      random_state=0).fit_transform(vectors)
    tries = range(1, min(max_groups, len(vectors) - 1) + 1)
    bic = {k: GaussianMixture(n_components=k, random_state=0)
              .fit(small).bic(small) for k in tries}
    best_k = min(bic, key=bic.get)          # lowest BIC wins
    gmm = GaussianMixture(n_components=best_k, random_state=0).fit(small)
    probs = gmm.predict_proba(small)        # one row per node, sums to 1
    groups = [np.where(probs[:, k] > threshold)[0].tolist()
              for k in range(best_k)]
    return [g for g in groups if g]

# Try it: 3 made-up topics, 40 fake embeddings each, 384 numbers long
rng = np.random.default_rng(7)
centres = rng.normal(size=(3, 384))
vectors = np.vstack([c + 0.35 * rng.normal(size=(40, 384)) for c in centres])
groups = group_nodes(vectors)
print(len(groups), "groups, sizes", sorted(len(g) for g in groups))
in_two = sum(1 for i in range(len(vectors)) if sum(i in g for g in groups) > 1)
print(in_two, "nodes in more than one group")

Install the libraries with pip install umap-learn scikit-learn numpy, then run the file. You should see two lines: 3 groups, sizes [40, 40, 40] and 0 nodes in more than one group.

Our runs took 40 seconds to nearly two minutes, depending on the machine. Much of that is one-time start-up: importing UMAP and its first fit took 29 to 43 seconds. A second fit in the same process took 0.3 seconds. UMAP also prints a warning that a fixed random_state turns off parallel threads; that is expected.

Notice the second line. Soft clustering allows a node in two groups, but in our synthetic runs no node passed the threshold twice. GMM probabilities are often close to 0 or 1, so count shared nodes on your own data before relying on them.

The collapsed tree search

The search step needs no special library. Here the vectors have only three numbers, standing for flood, money and land, so you can read the ranking by eye.

import numpy as np

def collapsed_search(levels, q, budget):
    pool = [node for level in levels for node in level]   # one flat pool
    def cosine(v):
        return float(np.dot(q, v) / (np.linalg.norm(q) * np.linalg.norm(v)))
    pool.sort(key=lambda node: cosine(node["vec"]), reverse=True)
    picked, used = [], 0
    for node in pool:
        if used + node["tokens"] > budget:
            break                                          # stop when full
        picked.append(node["name"])
        used += node["tokens"]
    return picked, used

# Toy vectors with 3 meanings: [flood, money, land]
chunks = [
    {"name": "chunk: 2019 flood level", "vec": [0.9, 0.0, 0.1], "tokens": 100},
    {"name": "chunk: culvert sizes", "vec": [0.8, 0.1, 0.0], "tokens": 100},
    {"name": "chunk: cost escalation", "vec": [0.1, 0.9, 0.0], "tokens": 100},
    {"name": "chunk: land acquisition", "vec": [0.0, 0.2, 0.9], "tokens": 100},
]
summaries = [
    {"name": "summary: flooding", "vec": [0.9, 0.2, 0.1], "tokens": 130},
    {"name": "summary: money risks", "vec": [0.2, 0.9, 0.2], "tokens": 130},
    {"name": "summary: all major risks", "vec": [0.6, 0.6, 0.5], "tokens": 130},
]
broad = [0.6, 0.6, 0.6]      # "What are the main risks?"
narrow = [0.85, 0.05, 0.0]   # "What size are the culverts?"
print(collapsed_search([chunks, summaries], broad, budget=300))
print(collapsed_search([chunks, summaries], narrow, budget=300))

Run it and you should see two lines. The broad question returns the all-risks and money-risks summaries using 260 tokens. The narrow question returns the culvert and flood-level chunks using 200 tokens.

The authors' reference code on GitHub uses different search defaults from the paper and has no way to update a tree in place. We read the code on its master branch, which showed 20 commits, on 19 October 2026.

Two rows change real results. With the default cap of 10 nodes, you get about half the paper's roughly 20. And the stop rule means the top of the tree usually holds up to 11 summaries, not one root.

The update row matters for the DPR archive. Calling add_documents on a tree that exists asks whether you meant add_to_existing, which is commented out. Answering yes changes nothing, and answering no rebuilds the whole tree. The question is a blocking input() call, which waits for a typed answer, so a job with its input closed stops with an EOFError, Python's end-of-input error.

The default embedding, summary and question-answering models call a paid hosted service. Swap in your own models through the config before you index anything private.

Explain it as plain chunk RAG plus a tree of summaries built offline, and justify it with a broad question that plain RAG fails. Use these steps as your answer frame.

1. 1

   Start with the failure

   Describe a long document and a broad question, like the risks across a 300-page report. Show that the top five chunks all come from one section.

2. 2

   Draw the offline build

   Chunk, embed, cluster, summarise, and repeat on the summaries. Mention soft clustering, which allows one chunk to sit under two topics.

3. 3

   Draw the online search

   Embed the question, rank all nodes in one pool, and fill a fixed token budget. Explain why mixing levels beats walking down the tree.

4. 4

   Add the guards

   Numbers must come from chunks, answers cite pages, and someone samples summaries for mistakes. Store a source version on every node.

5. 5

   Cover cost and updates

   Build cost grows in a straight line with length. Use one tree per document, so a change rebuilds one tree, not the whole archive.

6. 6

   Say how you would measure it

   Compare plain RAG and RAPTOR on a labelled set of broad and narrow questions. Keep RAPTOR only if broad answers improve and narrow ones hold steady.