Advanced RAG Explained: Naive vs Advanced vs Modular RAG

A simple RAG chatbot gives a wrong answer when its search brings back the wrong text, or misses the right text. The model can only answer from what the search hands it. Advanced RAG is the name for the fixes that target exactly these search failures.

Picture a common setup. An IT services firm in Hyderabad with about 5,000 employees builds an HR policy assistant. The setup is made up, but the problem is real.

The HR team keeps 220 policy documents on leave, travel, insurance and tax forms. Staff send hundreds of policy questions a week, and most answers already sit in those documents. An engineer builds the first version of the assistant in a week.

Before launch, the HR team collects 60 real questions from its inbox and checks every answer by hand. The assistant gets 38 right. The failures are not random. Three of them show the pattern:

• How many EL (earned leave) can I roll over to next year? The assistant says it cannot find the answer. The policy says earned leave can be carried forward, so the words never match.
• Where do I submit Form 12BB? The assistant explains general tax rules. The one page that names Form 12BB never comes back from the search.
• What is the carry-forward limit for earned leave? The assistant says 45 days, from an old 2023 policy. The current 2025 policy says 30 days, but it came back sixth, just outside the five chunks the model reads.

Each of these failures has a known fix. A 2023 survey of RAG research by Gao and colleagues sorts such fixes into three stages: naive RAG, advanced RAG and modular RAG. The term RAG itself comes from a 2020 paper by Lewis and colleagues. This guide follows the HR assistant through all three.

You need six words. Each one is a plain idea with a technical name, and the rest of the guide uses them without stopping.

• Chunk: a small piece of a document, usually a few paragraphs. The leave policy is split into 40 chunks, and one of them covers the carry-forward rule.
• Embedding: a list of numbers that captures the meaning of a piece of text. Texts with similar meaning get numbers that sit close together.
• Vector search: finding the chunks whose embeddings sit closest to the question's embedding. It searches by meaning, not by exact words.
• Retrieval: the search step that picks chunks for a question. The part of the program that does it is called the retriever.
• Top-k: how many chunks the retriever returns, where k is a number you choose. The first assistant uses k = 5.
• Context: all the text the model reads for one answer, sent as one prompt. It holds your instructions, the question and the retrieved chunks.

Naive RAG is the simplest version: search once with the user's exact question, take the top few chunks, and ask the model to answer from them. The survey calls it a Retrieve-Read process, one search and then one read.

Before any question arrives, naive RAG also indexes the documents. It cuts them into chunks, turns each chunk into an embedding and stores them. So the survey's full naive pipeline has three steps: indexing, retrieval and generation.

Picture a fresher who asks a hostel senior for notes on the heat chapter. The senior grabs the first five notebooks with heat on the cover and hands them over unread. Two are from an old syllabus, and the best notes, labelled thermodynamics, stay on the shelf.

Map it part by part. The shelf of notebooks is your chunk store. The word on the cover is the search match. The five notebooks are top-k = 5, and the fresher is the model, who studies whatever he is handed.

Here is the first version of the assistant. Follow the boxes from the question to the answer. The notes on the middle boxes show where the failed questions broke.

The 2023 survey lists the same three kinds of weak spots in naive RAG:

• Retrieval: low precision and recall. The search returns chunks that only look related, and misses chunks that matter.
• Generation: the model may add facts that are not in the retrieved text, often called hallucination.
• Augmentation, the step that joins chunks into one prompt: repeated or clashing chunks can make the answer disjointed or repetitive.

Advanced RAG keeps the same one-way pipeline from question to answer, but adds fixes before and after the search. The survey calls these two groups pre-retrieval and post-retrieval.

Pre-retrieval covers two kinds of change. Some improve the index, such as better chunk sizes or extra fields like a policy's year. Others improve the query, the text you search with, such as rewriting it in the documents' own words.

Post-retrieval covers changes to the search results before the model reads them. The survey names two: reranking, which reorders the chunks, and compression, which cuts them down.

This guide also adds hybrid search, keyword search running beside vector search, at the search step itself. The survey lists mixed retrieval as an index fix, and files hybrid retrieval under modular RAG's patterns. It fits here because it changes how the one search runs, not the shape of the flow.

Back at the hostel, a careful senior asks which professor and which exam first. She rephrases heat as thermodynamics, checks both covers and contents, and hands over only the two chapters that matter. That is rewriting, hybrid search, reranking and compression.

Here is the advanced pipeline. Pause and predict which failed question each new box fixes. The notes give the answer.

Query rewriting fixes searches that fail because the user's words differ from the document's words. A model rewrites the question into clearer search text before the search runs.

Employees type EL, roll over and next year. The policy says earned leave, carried forward and calendar year. So the rewrite step asks the model to restate each question in the style of an HR policy.

The rewritten query reads something like: earned leave carry-forward limit per calendar year. That now matches the right chunk, and the employee never had to learn HR language.

There are four common styles of rewriting, and each fits a different problem:

• Rewrite: turn a short or chatty question into one clear search query. Good for short forms like EL and WFH (work from home).
• Multi-query: write three or four versions of the question, search with each, and merge the results. Good when any single phrasing might miss.
• Sub-queries: split a two-part question into two searches. Good for comparing the Hyderabad and Pune travel rules.
• HyDE, short for hypothetical document embeddings: a model writes a made-up answer, and you search with that answer's embedding. A fake answer often sits closer to the real document than a short question does.

The HyDE paper accepts that the made-up answer may contain false details. Only its embedding is used, to find real documents that sit nearby. The employee's answer still comes from real chunks.

You combine them because each one misses what the other finds. Vector search finds text with similar meaning, and keyword search finds exact words, codes and names. Running both and merging the results is called hybrid search.

Vector search handles roll over versus carry forward well. A code like 12BB is different. An embedding model may never have learned what 12BB means, so meaning search can rank general tax pages above the one page that names the form.

Keyword search scores chunks by the exact words they share with the query. A standard scoring method is BM25. It gives more weight to rare words, so the one chunk that contains 12BB scores high.

We checked this with the rank-bm25 0.2.2 Python package on 7 toy HR chunks. 12bb appeared in 1 chunk and tax in 4. So 12bb weighed 1.466 against 0.303 for tax. For the query 12bb tax, the 12BB chunk scored 1.656 and the general tax page, which says tax four times, only 0.544.

The 0.303 is a floor, not the raw formula. A word in more than half the chunks gets a negative raw weight, and rank-bm25 0.2.2 raises it to a small positive value.

The 2024 modular RAG paper makes the same point: keyword retrievers help meaning-based ones with queries about rare entities. Rare codes such as policy numbers behave the same way.

To merge the two lists, a common method is reciprocal rank fusion. Each chunk earns points from its position in each list, so a chunk near the top of both lists wins. The engineer section shows it in ten lines of code.

Reranking fixes good chunks that come back in a bad order, and compression fixes context that is too long or too noisy. Both run after the search and before the model answers.

Reranking puts the best chunks first

A reranker reorders the candidate chunks. Some use simple rules, such as favouring variety, but the one here is a second, slower model that scores each chunk against the question. Vector search compared embeddings made separately for the question and for each chunk. It never looked at the two side by side.

One kind of reranker is a cross-encoder: a model that reads the question and the chunk together as one input and returns one relevance score. A 2019 paper by Nogueira and Cho used this setup to re-rank passages first found by BM25.

Reading every pair costs time, so you only rerank a short list. The assistant reranks its 30 merged chunks and keeps the best 4. The 2025 leave policy, which says 30 days, now comes first.

Context compression sends less, but better, text

Compression means cutting the retrieved text down to the parts that matter before the model reads it. You can drop chunks the reranker scored low, or keep only the sentences that relate to the question.

More text is not always better. A 2023 study called Lost in the Middle tested two tasks: answering from many documents, and finding a value by its key in a long list. Models often did best when the needed fact sat at the start or end of the input.

When the fact sat in the middle, results dropped sharply, even for models built for long inputs. So four short, relevant extracts can beat fifteen long chunks. Shorter context also costs less when your model provider bills by the token.

A token is a small piece of text, often a word or part of a word. Fewer tokens in means a smaller bill for every question.

Modular RAG treats each step of the pipeline as a separate part, or module, that you can add, swap or rearrange. Instead of one fixed line, the flow can branch into different paths and loop back to search again.

A module is one part with one job and a clear input and output, like the reranker or the keyword search. A 2024 paper by four of the survey's authors develops this idea, and its title compares the parts to toy building bricks.

Its abstract names four flow patterns, and the body adds a fifth, tuning, which trains the retriever or the answering model further:

• Linear: modules run in a fixed line. Naive RAG is the simplest linear flow, and the advanced pipeline above is linear too, with pre-retrieval and post-retrieval steps added.
• Conditional: a routing module picks one pipeline for each question.
• Branching: several searches or paths run side by side, and their results are merged.
• Looping: retrieval and generation repeat, and a judge step decides when to stop.

The HR assistant needed this soon after launch, because some questions had nothing to do with documents. One employee asked how many leave days she had left. That answer lives in the HR database, not in any policy. A simple thanks needs no search at all.

Sending both through the policy pipeline wasted model calls and produced odd answers. So the team added a router: a step that labels each question and sends it down the right path. Each path can change without touching the others.

Here is the modular assistant. Follow each labelled arrow from the question. On the policy path, an answer not backed by the policy text gets one more rewrite and search.

Start with naive RAG, measure it on real questions, and upgrade only where you can name the failure. A few advanced fixes are often enough. Add routing and loops only when your questions need different sources or a second try.

Here is the full map of upgrades, each tied to the problem it fixed in the HR assistant.

Here is how the three versions compared on the 60-question test set.

Read the table as a trade. Each step bought correct answers with time and money. The retry loop caught most of the modular gain, and leave balance questions now get real answers from the HR database.

When an answer is wrong, the chart below helps you pick the fix. Start with one failed question and ask where the right chunk ended up.

Run the failed questions through it. The EL and Form 12BB questions failed the top-20 check, so they needed rewriting and hybrid search. The carry-forward question was found but ranked sixth, so it needed reranking. The leave balance question needed a router.

The team followed this order to upgrade the assistant:

1. 1

   Build the naive baseline

   Get the simplest version working end to end, with one search and k = 5.

2. 2

   Build a test set

   Collect real questions with checked answers, like the 60 here, before changing anything.

3. 3

   Sort the failures

   For each wrong answer, note whether the right chunk was missing, ranked too low, or not in the documents at all.

4. 4

   Fix the biggest group first

   Add the one upgrade that matches the most failures, then rerun the whole test set.

5. 5

   Keep only what pays

   Keep an upgrade only if the score rises enough to justify the extra cost and wait.

Reciprocal rank fusion (RRF) gives each chunk 1 divided by (60 plus its rank) in each list, then adds the points up. It uses only positions, never the raw scores, which is why it suits hybrid search.

Raw scores do not mix. In our BM25 test the top score was 4.116. A cosine similarity, a common vector score, always sits between minus 1 and 1. Adding the two would let the keyword score drown the vector score. The 2009 RRF paper by Cormack, Clarke and Buettcher combines ranks for this reason, with the constant 60 fixed in a pilot study.

Here is RRF in plain Python 3, with no packages. It merges the two result lists for the Form 12BB question. Save it as rrf.py and run python3 rrf.py.

def rrf(*ranked_lists, k=60):
    """Reciprocal rank fusion: merge ranked lists using ranks only."""
    scores = {}
    for ranking in ranked_lists:
        for rank, chunk_id in enumerate(ranking, start=1):
            scores[chunk_id] = scores.get(chunk_id, 0) + 1 / (k + rank)
    return sorted(scores.items(), key=lambda item: item[1], reverse=True)

by_words = ["tax-12bb", "tax-80c", "tax-hra"]        # keyword search, best first
by_meaning = ["tax-general", "tax-hra", "tax-12bb"]  # vector search, best first

for chunk_id, score in rrf(by_words, by_meaning):
    print(chunk_id, round(score, 4))

You should see exactly these four lines:

tax-12bb 0.0323
tax-hra 0.032
tax-general 0.0164
tax-80c 0.0161

The constant matters. A smaller one rewards the very top ranks more. In our run, rank 1 earned twice the points of rank 3 at k = 1, but only 1.03 times at k = 60.

In the paper's pilot runs, 60 was near-optimal but the choice was not critical: values from 10 to 100 scored almost the same. Keep 60 unless your test set shows better, and log both input lists.

Compare where the right chunk landed with what the model actually read. If the chunk is missing, fix the search. If it is found but too low, add reranking. If the model read it and still erred, fix the context or prompt.

This needs one extra field in your test set: the ID of the chunk that holds each answer. Here is the decision chart as code, run on the four failures from this guide. It also runs on plain Python 3.

def stage_that_broke(right_id, top_20, read_k=5):
    """Say which stage failed for one wrong answer."""
    if right_id is None:
        return "source: no document has it, so route it elsewhere"
    if right_id not in top_20:
        return "search: rewrite the query or add keyword search"
    if top_20.index(right_id) >= read_k:
        return "ranking: add a reranker"
    return "reading: compress the context or fix the prompt"

failures = [  # question, chunk that holds the answer, what the search returned
    ("How many EL can I roll over?", "leave-2025",
     ["holiday-list", "leave-sick", "wfh-rules", "leave-2023"]),
    ("Where do I submit Form 12BB?", "tax-12bb",
     ["tax-general", "tax-80c", "tax-hra"]),
    ("What is the carry-forward limit?", "leave-2025",
     ["leave-2023", "leave-faq", "holiday-list", "leave-sick", "wfh-rules", "leave-2025"]),
    ("How many leave days do I have left?", None,
     ["leave-2025", "leave-faq"]),
]
for question, right_id, top_20 in failures:
    print(question, "->", stage_that_broke(right_id, top_20))

Expected output:

How many EL can I roll over? -> search: rewrite the query or add keyword search
Where do I submit Form 12BB? -> search: rewrite the query or add keyword search
What is the carry-forward limit? -> ranking: add a reranker
How many leave days do I have left? -> source: no document has it, so route it elsewhere

Count the labels across your whole test set before building anything. If most failures say ranking, a reranker is your next step. If most say search, a reranker cannot help, because it only reorders what the search found.

Map each label to a function in a plain dictionary, fall back to a safe default for unknown labels, and cap retries with a fixed loop count. Here is a runnable version with stand-ins for the model calls.

MAX_SEARCHES = 2  # the first search plus one retry, never more

POLICY_INDEX = {"earned leave carry forward limit": "Up to 30 days (Leave Policy 2025)"}
REWRITES = {"How many EL can I roll over?": "earned leave carry forward limit"}

def search_policies(query):          # stand-in for the advanced pipeline
    return POLICY_INDEX.get(query)

def rewrite(query):                  # stand-in for a model call
    return REWRITES.get(query, query)

def policy_path(question, employee_id):
    query = question
    for _ in range(MAX_SEARCHES):
        answer = search_policies(query)
        if answer:                   # stand-in for the "backed by the text?" check
            return answer
        query = rewrite(query)
    return "No policy covers this. Please ask HR."

def records_path(question, employee_id):
    return f"Employee {employee_id}: 12 days of earned leave left"

def chat_path(question, employee_id):
    return "Happy to help."

MODULES = {"policy": policy_path, "my_data": records_path, "chat": chat_path}

def handle(question, employee_id, label):
    """label comes from a small model call; employee_id comes from the login."""
    module = MODULES.get(label.strip().lower(), policy_path)
    return module(question, employee_id)

print(handle("How many EL can I roll over?", "E1042", "policy"))
print(handle("How many leave days do I have left?", "E1042", "my_data"))
print(handle("Thanks!", "E1042", " Chat"))
print(handle("Is Diwali a holiday?", "E1042", "not sure"))

Run it with python3 and you should see:

Up to 30 days (Leave Policy 2025)
Employee E1042: 12 days of earned leave left
Happy to help.
No policy covers this. Please ask HR.

Three design choices carry the safety. The model only picks a label, and an unknown label lands on the policy path. The employee ID comes from the login, so the model's label cannot point the lookup at someone else's records. And range(MAX_SEARCHES) makes an endless loop impossible.

Adding a payslip module later means one new dictionary entry. Swapping the reranker inside the policy path does not touch the router at all. That is what modular means in practice.

Not always where blog posts do. The 2023 survey and the 2024 modular RAG paper place some fixes in different stages, so treat the three labels as a map, not a law.

Two things follow. First, an interviewer who files hybrid search under modular RAG is quoting the survey, not making a mistake. Second, the modular paper calls a pipeline with pre-retrieval and post-retrieval steps linear, so advanced RAG is one linear flow inside modular RAG.

Explain it as a series of fixes, each tied to a failure you can measure. Interviewers want to hear why you add each part, not a list of technique names.

1. 1

   Draw the naive baseline

   Question, search, top chunks, model, answer. Say it is the starting point, not the final design.

2. 2

   Say how you would measure it

   Describe a test set of real questions, each with its checked answer and the chunk that holds it, run after every change.

3. 3

   Map failures to stages

   Wording gaps go to query rewriting, missed codes to hybrid search, and bad order or noise to reranking and compression.

4. 4

   Add modules where sources differ

   Add a router for questions that need a database or no search, and a loop with a hard limit.

5. 5

   Talk about cost and wait time

   Each upgrade adds model calls or a reranker. Give rough numbers, and say they are estimates to check on the test set.