Agentic RAG: When Retrieval Becomes a Decision

A RAG chatbot, one that looks up documents before it answers, often goes wrong because it searches once, in one place, and trusts whatever comes back. Nothing in it stops to ask whether the results fit the question, or whether they are still current. Agentic RAG adds those missing decisions.

Picture a common setup. A university runs a helpdesk chatbot for students, answering questions about fees, exams, hostels and scholarships from its own documents. The setup is made up, but the problem is real.

The first version uses classic RAG. Every question goes through the same three steps: search the document store, take the five closest passages, and ask the model to answer from them. The model here is a large language model, the kind of AI that reads and writes text.

In admission season the bot handles about 3,000 questions a day, at about Rs 0.30 and 2 seconds each. Most answers are fine. Then the registrar's office reads 200 saved chats and finds three kinds of trouble.

• A student types a polite thank you. The bot still searches, then replies with a paragraph about library fines.
• A student asks about the MTech fee and the north-east scholarship in one message. The fee lives in a fee table and the scholarship rules live in circulars, but the bot searched only once.
• A student asks whether scholarship holders still get a free hostel. The search finds a 2024-25 circular that says yes. A newer circular for 2026-27 cut the waiver to half, and the bot never saw it.

The last one hurts. The office gets 40 upset emails in one week from families who planned around a full waiver. Classic RAG could not have caught it, because it has no step where anything gets decided.

You need five words: chunk, source, query, tool and sufficiency check. Each gets a plain meaning and an example from the helpdesk.

Chunk. Documents are split into small pieces called chunks before they are searched. Example: the bot pastes five handbook chunks above the student's question.

Source. A source is one place the system can look for facts. The helpdesk has three: the academic handbook, the registrar's circulars, and a fee table in the university database.

Query. A query is the text you actually search with, not always the student's exact words. Example: a long, chatty message becomes the query "scholarship hostel fee 2026-27".

Tool. A tool is an action the model may ask for by name, such as search_circulars. Your code runs it and hands back the result. An agent is a program where the model picks the next tool itself, in a loop.

Sufficiency check. A sufficiency check is a step that asks whether the results really answer the question. Example: the circular found is from the 2024-25 session, so the answer is no, and the system searches again.

Classic RAG searches the same way every time because the steps are fixed in your code. The model only writes the answer at the end, from whatever the search returned.

Follow the arrows from the question to the reply. Notice that the model appears in only one box, right at the end.

This is a workflow: a program where you write every step in advance. That is a real strength. You can test each step on its own, every question costs about the same, and replies are quick.

The weakness is just as clear. The pipeline cannot say "skip the search", "look in the fee table instead" or "that circular is old, try again". Those are decisions, and classic RAG has nobody to make them.

Some RAG pipelines add fixed extra steps, such as rewriting every query or reranking, re-sorting the results with a second, slower scorer. They are still classic in this sense, because the same steps run for every question.

Agentic RAG is RAG where retrieval becomes a decision the model makes in a loop, instead of a fixed step your code always runs. The model chooses whether to search, which source, what query, and when to stop. Your code still runs every search and sets the limits.

A 2025 survey of agentic RAG by Singh and colleagues describes the same shift. It says traditional RAG is held back by static workflows that lack the adaptability that multi-step reasoning needs. It names four agent patterns: reflection (checking its own work), planning, tool use and multi-agent collaboration (several agents splitting the work).

A picture from the record room

Think of the university's record room, full of steel almirahs packed with files. Classic RAG is a new junior clerk who, for any question, brings back the five files that look most similar. He brings five files for a thank you too, and never checks their dates.

Agentic RAG is a senior office assistant. For a thank you, she just replies. For a fee question, she opens the fee register, not the rule book. If a circular looks old, she goes back for the newer one.

Here is the loop the team builds. Start at the question and follow the arrow from the check back to the round limit. That arrow is what lets the helpdesk look again.

Now take the question that mixes two problems from the office's list. An MTech student asks for the fee and whether the north-east scholarship still gives a free hostel. Pause and predict: how many searches will the loop need before it can answer?

1. 1

   Round 1: get the fee

   The model sees two parts, a fee and a scholarship rule. It calls get_fee_table for MTech: tuition Rs 48,500 and hostel Rs 14,000 a semester. The check says partly answered.

2. 2

   Round 2: search the circulars

   It calls search_circulars with "north-east scholarship hostel fee". The best match is the 2024-25 circular saying scholarship holders pay no hostel fee.

3. 3

   The check says no

   The circular is not from the 2026-27 session, so code rejects it before the model judges anything. The note says to search again with the session year.

4. 4

   Round 3: search again

   The model rewords the query and adds 2026-27. It finds the 2026-27 circular: scholarship holders now pay half the hostel fee. The check says enough.

5. 5

   Round 4: answer

   The model answers with both facts and names the 2026-27 circular, so the student can read it.

An agentic RAG system makes four decisions that classic RAG hard-codes: whether to search, where to search, what to search for, and whether the results are enough. Each one is a chance to save effort, and a chance to make a mistake.

Whether to search at all

Some messages need no lookup. Greetings, thanks and questions about what the bot can do can be answered directly. Skipping the search saves time and stops the bot quoting library fines at a polite student.

The risk runs the other way too. A model may skip a search it needed and answer a fee question from memory. That memory comes from its training data, the text it learned from before release, which knows nothing about this university's fees.

Where to search

Choosing a source means picking the right lookup for the kind of fact. A fee is a number in a table, so a database lookup beats searching text. A standing rule lives in the handbook, and a recent change lives in a circular.

Each source is a tool with a short description, and the model reads those descriptions to choose. Vague descriptions lead straight to wrong choices, so write them like a label on a file: what is inside and when to use it.

What to search for and when to stop

The student's own words are often a poor query. An agent can reword the query, split a two-part question into two searches, or add a filter such as the session year. It also decides when the evidence is enough, which the engineer sections below show in code.

Classic RAG runs the same fixed search for every question, while agentic RAG decides how many searches to run, where, and with what query. The table shows how that one change affects everything else.

Read the searches row carefully. The answer also takes a round, so a run that answers makes at most four searches. A run that spends all five rounds searching is handed over to people.

Classic RAG is better when most questions are simple, one source holds the answers, and speed or cost matters more than rare tricky cases. Start with it, and measure where it fails before you add decisions.

These signs say classic RAG is enough:

• Questions are short lookups from one place, like the library's closing time.
• Documents rarely change, so an old result is not a big risk.
• Users need an answer in a second or two.
• Your team cannot yet test and read whole runs, step by step.

These signs point toward agentic RAG:

• Questions often have two parts that live in different sources.
• Old and new versions of the same document both sit in the store.
• Wrong answers are costly, like fee amounts that families plan around.
• A second search is much cheaper than a wrong answer going out.

You rarely have to pick one for everything. The team adds a router: a step that labels each question and sends it down the right path. In the diagram below, notice that only the agentic path can reach the fee table and the registrar's office.

The router is a decision too, and it can be wrong. The team logs every label, and the registrar's office checks a sample each week.

You build it as a for loop with a round limit. Each round asks the model for the next step, runs the chosen tool, and appends the results and the check's verdict to a notes list. When the rounds run out, it hands over to a person.

The first block is the real logic. The second block holds stand-ins for the model and the search, so the file runs offline. Paste both into one file, first block on top, and run it; we ran it on Python 3.13 with no packages.

CURRENT_SESSION = "2026-27"

def check_results(question, new_results, evidence):
    fresh = [p for p in new_results if p.session in (CURRENT_SESSION, "all")]
    if not fresh:                          # layer 1: plain code checks the date
        return "Check: nothing from 2026-27. Search again with the session."
    evidence.extend(fresh)                 # only checked passages are kept
    reply = ask_judge(question, evidence)  # layer 2: a model judges the rest
    verdict = reply.strip().lower().rstrip(".!")
    if verdict == "yes":
        return "Check: enough. Answer now and name the sources."
    if verdict == "partly":
        return "Check: partly answered. Search for the missing part."
    return "Check: not relevant. Try another source or query."

def reply_to_student(question, max_rounds=5):
    notes, evidence, tried = [question], [], set()
    for round_no in range(1, max_rounds + 1):
        name, arg = next_step(question, notes, evidence)
        print(f"Round {round_no}: {name}")
        if name == "answer":
            return arg
        if name not in TOOLS or (name, arg) in tried:
            notes.append(f"Error: unknown tool or repeated query: {name} {arg}")
            continue
        tried.add((name, arg))
        results = TOOLS[name](arg)
        notes.append(results)
        notes.append(check_results(question, results, evidence))
        print("   ", notes[-1])
    return "Out of rounds: handed to the registrar's office."

In a real system, next_step sends the question, the notes and the tool descriptions to your model and reads back its choice. ask_judge sends the question and the evidence and reads back one word. Here both are plain functions that play the model's part.

from dataclasses import dataclass

@dataclass
class Passage:
    text: str
    session: str    # "2024-25", "2026-27", or "all" for rules that never expire

CIRCULARS = [
    Passage("North-east scholarship holders pay no hostel fee.", "2024-25"),
    Passage("From 2026-27, scholarship holders pay half the hostel fee.", "2026-27"),
]
FEES = {"MTech": Passage("MTech: tuition Rs 48,500, hostel Rs 14,000 a semester.", "2026-27")}

def words(text):
    return set(text.lower().replace(",", " ").replace(".", " ").split())

def search_circulars(query):    # stand-in search: the one best keyword match
    return [max(CIRCULARS, key=lambda p: len(words(query) & words(p.text)))]

def get_fee_table(programme):
    return [FEES[programme]]

TOOLS = {"search_circulars": search_circulars, "get_fee_table": get_fee_table}

def next_step(question, notes, evidence):    # stand-in for the model's choice
    if "Answer now" in notes[-1]:
        return "answer", " | ".join(p.text for p in evidence)
    if not evidence:
        return "get_fee_table", "MTech"
    if "Search again" in notes[-1]:
        return "search_circulars", "scholarship hostel fee 2026-27"
    return "search_circulars", "north-east scholarship hostel fee"

def ask_judge(question, passages):    # stand-in for the model's verdict
    text = " ".join(p.text for p in passages)
    return "Yes." if "MTech" in text and "half" in text else "Partly"

print(reply_to_student("What is the MTech fee, and is hostel still free on "
                       "the north-east scholarship?"))

Run the file and you should see exactly these eight lines:

Round 1: get_fee_table
    Check: partly answered. Search for the missing part.
Round 2: search_circulars
    Check: nothing from 2026-27. Search again with the session.
Round 3: search_circulars
    Check: enough. Answer now and name the sources.
Round 4: answer
MTech: tuition Rs 48,500, hostel Rs 14,000 a semester. | From 2026-27, scholarship holders pay half the hostel fee.

Three details matter. The tried set refuses to run a repeated query, so a stuck model wastes rounds but not searches. In our test, a model that kept sending one query ran one search, then hit the five-round limit and was handed over.

An unknown tool name is turned into an error note, not a crash. And every tool only reads, so a bad choice wastes a round but never changes data.

Check it yourself: call reply_to_student with max_rounds=2. You should see rounds 1 and 2, then "Out of rounds: handed to the registrar's office."

It runs a sufficiency check after each search, asking whether the results are current and whether all the evidence so far answers the question. If not, the loop searches again, or hands over when it runs out of rounds.

The check in the first block has two layers. First, plain code rejects passages from the wrong session, without asking the model. Then a model judges the harder part: do the checked passages fully answer what was asked?

Parse the verdict defensively

Models often add punctuation. Our first version compared the reply with "yes" after only trimming spaces and lower-casing it. In our test, a reply of "Yes." then fell through to the last branch and counted as not relevant.

The fixed line also strips trailing full stops and exclamation marks. Replies like "Yes, mostly." still count as no, which is the safe default: the loop searches again rather than answering early. So does a bold "Yes" or a "Yes" with a reason on the next line, so ask for one word and log any other reply.

Four well-known papers each hand a different decision to a model or a small trained helper. A token, used in the table, is a small piece of text, often a word or part of one. None of them is a full helpdesk loop like the one above, so borrow the idea, not the numbers.

In FLARE's main version, low confidence means the model gave some token in its draft a low probability. The query is the draft itself, with the unsure tokens masked out or turned into questions.

Self-RAG uses a separate critic model only while preparing training data, to insert the reflection tokens. At answer time the trained model decides on its own. That moves the decisions into the model, instead of a loop in your code.

Corrective RAG's evaluator has about 0.77 billion parameters, the learned numbers inside a model, and the authors call it lightweight. It scores each document from -1 to 1. The authors set the two thresholds separately for each test dataset, which is a hint that you will need to tune your own.

What Adaptive-RAG shows about routers

Adaptive-RAG is the closest to the helpdesk router. Its classifier has 770 million parameters, and its training labels are made automatically, from which strategy answered correctly and from the type of test dataset.

The classifier is far from perfect. In the paper's Table 4, it picked the right label for 54.52 percent of questions overall. The abstract still claims better overall efficiency and accuracy.

Their Table 1 shows what that means. Across three answering models, the router came within 2 percentage points of the accuracy of always searching step by step, in 41 to 53 percent of its time. It was still slower than one search for every question.

Their Table 3 shows why routing pays off. Questions labelled for no retrieval took about 0.35 seconds each, one search about 3.08 seconds, and step-by-step retrieval about 27.18 seconds. Appendix A.3 of the paper's PDF says the runs used Nvidia A100 GPUs with 80 GB of memory. Your times will differ, so compare the ratios, not the seconds.

Agentic RAG usually costs more and takes longer than classic RAG, because every extra round adds a model call and a search. How much more depends on your questions, so measure it on real traffic.

Model services usually charge by tokens going in and out. In a loop like ours, each round sends the growing notes back to the model, on top of the instructions and tool descriptions.

Take made-up sizes: 60 tokens for the question and 520 per round of results and verdict. By round 5, the question and notes come to 2,140 tokens, against 60 in round 1. The instructions and tool descriptions are added on top, every round.

Now the daily maths, with the table's made-up costs. Classic RAG on 3,000 questions costs Rs 900 a day. Sending every question through the loop costs about Rs 2,700.

With the router, say 450 questions are chit-chat, 1,800 simple and 750 hard. Hard questions use more rounds than average, so say they cost Rs 1.40 each, not Rs 0.90 (all made up).

That is about twice plain classic RAG and about two-thirds of sending everything through the loop. The extra Rs 885 a day buys a second look at the hard questions, where wrong answers cost the most.

Waiting time matters even more on a chat screen. Two seconds feels quick, and twelve feels broken. The team shows short status lines, like "Checking circulars", so a slow answer does not look like a crash.

• Cap the rounds, so a confused run cannot keep spending.
• Cap the spend per question in code, and hand over when it is reached.
• Use a small, cheap model for the check if it judges well enough on your sample questions.
• Log rounds and cost per question, so you notice when the average starts to creep up.

Agentic RAG adds new failures because the model now makes choices. It can search too little, too much, in circles, in the wrong place, or trust a bad document. Each failure needs a guard in your code.

• Searching too little. It answers a fee question from memory. Guard: reject any fee, date or rule answer that names no checked document.
• Searching too much. It runs four searches where one would do. Guard: the round limit, plus a count of searches per question in your logs.
• Going in circles. It repeats the same query. Guard: the tried set, which turns a repeat into an error note.
• Picking the wrong source. It searches the handbook for a fee. Guard: clear tool descriptions, and tests that check which tool was used.
• Stopping too early. It accepts the 2024-25 circular. Guard: the session check in code, which the model cannot argue past.
• Trusting a planted instruction. A document hides text written to steer the model. Guard: read-only tools, and retrieved text treated as data, never orders.

The last one deserves care. OWASP, a non-profit that publishes security guidance, lists prompt injection first in its Top 10 for LLM Applications, 2025 edition, the latest as of 19 October 2026. LLM applications are apps built on large language models.

OWASP calls this case indirect prompt injection: the model takes in outside content, such as websites or files, that changes its behaviour. Its advice includes the least access the job needs, marking untrusted content clearly, and a person approving risky actions. OWASP also says it is unclear whether any method fully prevents prompt injection, so these steps limit the damage rather than remove it.

Permissions need the same care. Because the model now chooses sources, the rule about who may see what must live inside each tool. The tool should use the logged-in student's id from your code, never an id the model typed.

Finally, agentic RAG is harder to debug, because two similar questions can take different paths. So the team saves every round: the query, the source, the results and the check's verdict.

Explain it as classic RAG plus a few decisions, and justify each decision with a failure it fixes. The steps below give a frame you can follow on a whiteboard.

1. 1

   Start with classic RAG

   Draw the fixed pipeline first: question, one search, top results, answer. Say that it is cheap, fast and easy to test.

2. 2

   Show where it fails

   Give concrete failures, like a two-part question across two sources, or an old circular beating a new one.

3. 3

   Add only the decisions you need

   Name which choices you hand to the model: skip the search, choose a source, reword the query, or judge the results.

4. 4

   Put the limits in code

   A round limit, a spend limit, the session check, read-only tools, and permission checks inside each tool.

5. 5

   Add a router

   Send simple questions down the classic path and only the hard ones into the loop. Say the router can mislabel, and how you would sample its labels.

6. 6

   Estimate, then measure

   Do rough cost and wait-time maths out loud. Then describe a set of sample questions with known answers, run before every change.