How to Debug LLM Hallucination: Prompt, Context or Model?

You fix the wrong layer, so the bug survives and comes back with the next customer. Debugging LLM hallucination well means finding where the wrong answer started before you change anything.

Picture a common setup. A health insurance company in Pune runs a help bot that answers about 3,000 policy questions a day from its brochures. The setup is made up, but the problem is real.

A customer on the Silver plan bought her policy 14 months ago. Her doctor says she needs a knee replacement costing about Rs 2.4 lakh (240,000 rupees). She asks the bot: "Is my knee replacement surgery covered?"

The bot replies: "Yes, knee replacement is covered up to Rs 3 lakh. You can go ahead with the surgery." At the hospital, her claim is rejected. Her plan makes her wait 24 months for joint replacement.

A waiting period is a fixed time after you buy a policy during which some treatments are not covered. In this policy, Silver makes you wait 24 months for joint replacement, and Gold makes you wait 12.

She pays the hospital herself and complains. The engineer who looks after the bot is asked why the AI made this up. "It made it up" is a guess, not a finding, and the rest of this guide finds the real cause.

Seven words carry the rest of this guide. Each one gets a plain meaning and an example from the insurance bot.

• Hallucination: an answer that sounds right but is not supported by real-world facts or by the text the model was given. The bot's "covered up to Rs 3 lakh" is one.
• Prompt: the full text sent to the model, with instructions first, then any documents, then the user's question.
• Trace: the saved record of one request, step by step. It shows what came in, what was looked up, what was sent to the model and what came back. This chat's trace has the ID req-7f3a91.
• Retrieval: the bot searches its own documents and pastes the best matches into the prompt. Each pasted piece is a chunk. The setup is called RAG, short for retrieval-augmented generation.
• Tool and tool result: a tool is a function your code runs to fetch facts, and the tool result is what it returns. Here get_policy returns the customer's plan and start date.
• Grounded: an answer is grounded when every claim in it traces back to a source you gave the model. A citation is the pointer naming that source.
• Regression set: saved test cases you run before every change, so old bugs cannot quietly return.

Reproduce the failure, read its trace, then check each layer from the outside in, and question the model last. Each check either finds the cause or rules one layer out.

Think of a ceiling fan that stops at home. Before you replace the fan, you check the switch, then the trip switch in the fuse box, then the wiring. The fan comes last because it costs the most to change.

The mapping is direct. The switch is the user's input, the fuse box is retrieval, the wiring is the prompt, tools and parsing, and the fan is the model.

Where the picture breaks: a fan either spins or it does not. A model can answer the same input correctly 3 times and wrongly 7 times, so you count runs instead of looking once.

1. 1

   Reproduce

   Make the wrong answer happen again with the same inputs, several times. If it will not show up, run it more times (see the rerun section) before you guess at a cause.

2. 2

   Read the full trace

   Look at every step of the request, not just the final reply.

3. 3

   Check inputs and retrieved context

   Did the user's message and the pasted chunks contain the facts needed for a correct answer?

4. 4

   Check the prompt

   Do the instructions push the model toward a confident answer, or leave out an important rule?

5. 5

   Check tool results

   Did each tool return correct data, or an error that your code hid?

6. 6

   Check output parsing

   Did your code read the model's reply correctly before showing it?

7. 7

   Suspect the model

   Only when the facts and instructions were right, and the answer was still wrong, is the model the main cause.

Send the same input again several times and count the wrong answers. Then read the trace of the original request from start to end.

One run proves little, because model output has randomness. A setting called temperature controls how much: a higher temperature lets the model pick less likely words (strictly, tokens: words or word pieces) more often. Setting it to 0 reduces the randomness, but it does not guarantee identical answers.

A 2024 study set five models up to be repeatable. It used temperature 0, a fixed seed (a number that pins the random choices) and top-p, a second randomness setting, at 1. Over 10 runs of eight tasks, accuracy still varied by up to 15%.

The authors name speed tricks inside model servers, such as batching many users' requests together, as a possible cause. For hosted models, they say, they can only speculate.

So the engineer replays the request 10 times with the same prompt and chunks. The wrong "covered" answer comes back 9 times out of 10 (made up). That is a real, repeatable failure.

Next comes the trace for req-7f3a91, read top to bottom. The final reply only shows the symptom. The trace shows six steps: question in, get_policy ran, search ran, prompt built, model replied, reply parsed and shown.

Without saved traces you are guessing. Setting them up is its own topic, covered in the observability guide linked at the end.

Print the exact chunks and tool results that went into the prompt, and check whether the facts needed for a correct answer were there. If they never reached the model, it could not have answered correctly.

The customer's message was clear: knee replacement, her own policy. The engineer checks the get_policy tool a step early, because its plan is needed to judge the chunks. It returned Silver and a start date 14 months back, which is correct.

This short script prints each chunk in the saved trace and flags any chunk about a plan the customer does not hold. The trace is typed into the script, so it runs with plain Python 3. Every output shown in this guide came from Python 3.13.

# A saved trace, typed in here so the script runs on its own.
# In your app it would come from your trace store.
trace = {
    "id": "req-7f3a91",
    "get_policy": {"plan": "Silver", "start": "2025-08-05"},
    "chunks": [  # the text pasted into the prompt, best match first
        {"id": "gold-knee", "plan": "Gold", "score": 0.91},
        {"id": "gold-waiting", "plan": "Gold", "score": 0.88},
        {"id": "claims-howto", "plan": "All plans", "score": 0.74},
        {"id": "gold-limits", "plan": "Gold", "score": 0.71},
    ],
}

plan = trace["get_policy"]["plan"]
print("Customer plan:", plan)
for rank, chunk in enumerate(trace["chunks"], start=1):
    wrong = chunk["plan"] not in (plan, "All plans")
    print(rank, chunk["id"], chunk["plan"], "OTHER PLAN" if wrong else "ok")
print("silver-waiting in prompt:",
      any(c["id"] == "silver-waiting" for c in trace["chunks"]))

# Expected output:
# Customer plan: Silver
# 1 gold-knee Gold OTHER PLAN
# 2 gold-waiting Gold OTHER PLAN
# 3 claims-howto All plans ok
# 4 gold-limits Gold OTHER PLAN
# silver-waiting in prompt: False

What just happened? Three of the four chunks came from the Gold brochure. The one Silver rule that mattered, silver-waiting, never reached the prompt. The full gold-knee chunk in the trace holds the Rs 3 lakh line word for word.

Why did the search miss it? It matched the words of her question. The Gold brochure says "knee replacement", while the Silver brochure says "joint replacement", so Gold scored higher. Nothing in the question told the search which plan she holds.

So the model did not invent "covered up to Rs 3 lakh". It repeated a real sentence from the wrong document. The prompt did name her plan as Silver, but nothing asked the model to compare that with the chunks.

The root cause, meaning the first thing that went wrong, is retrieval. The model's slip comes second.

Read the full prompt exactly as the model saw it, and the raw reply before your code touched it. Bugs hide in what the instructions said and in how your code read the answer.

Checking the prompt

The instruction part of the prompt is called the system prompt. This bot's system prompt said: "Be helpful and always give the customer a clear yes or no."

That line is a second problem. It pushes the model to answer even when its sources do not match the customer's plan. Nothing allowed it to say "I am not sure" or hand the chat to a person.

The engineer tests this with one controlled change: change one thing, keep everything else the same, and run again. Same wrong chunks, plus one new line: if the sources are not about the customer's plan, say you are not sure.

Now the bot says "not sure" in 8 runs out of 10 (made up). The prompt was a contributing cause, not the root cause: the right chunk is still missing.

Checking output parsing

Output parsing is the code that turns the model's raw text into data your app uses. This bot asks for JSON (a simple text format for data) with a covered field, and shows a green Covered badge when covered is true.

Here the raw reply said covered: true, and the app showed exactly that, so parsing is cleared. In other bots this layer is a real suspect. Here is a classic Python slip and a strict parser that stops it.

import json

raw = '{"covered": "false"}'   # the model wrote false as text, in quotes
print(bool(json.loads(raw)["covered"]))   # what a loose check sees

def parse_covered(raw):
    data = json.loads(raw)   # raises an error if the JSON is broken
    if not isinstance(data.get("covered"), bool):
        raise ValueError(f"covered must be true or false, got {data.get('covered')!r}")
    return data["covered"]

for raw in ['{"covered": false}', '{"covered": "false"}', '{"covered": fals']:
    try:
        print("OK  ", parse_covered(raw))
    except ValueError as err:   # a JSON parse error is also a ValueError
        print("STOP", type(err).__name__, "-", err)

# Expected output:
# True
# OK   False
# STOP ValueError - covered must be true or false, got 'false'
# STOP JSONDecodeError - Expecting value: line 1 column 13 (char 12)

What just happened? In Python, any non-empty text counts as true, so the word "false" in quotes would light the green badge. The strict parser refuses the wrong type and broken JSON instead of guessing.

Python's JSON error is a kind of ValueError (Python's error for an input with a bad value), which is why one except line catches both. Three more parsing bugs to look for:

• Falling back to a default such as covered = true when the JSON fails to parse.
• Cutting off a long reply at a length limit, so "covered after the waiting period ends" becomes "covered".
• Keeping only the first sentence, which drops a warning the model added after it.

It is the model's fault when it had the right facts and clear instructions and still answered wrongly, run after run. You prove that with a control test: give it perfect inputs and see whether it still fails.

The engineer builds the prompt by hand with the right Silver chunk and the policy start date. Same model, same instructions. The model answers "not covered yet, about 10 months of the waiting period are left" in 10 runs out of 10 (made up).

That settles it. Swapping in a bigger, costlier model is not the fix here. It might notice the plan mismatch more often, but the Silver rule would still be missing from its prompt.

Back to the fan: the fan was fine, and the fuse box was feeding the wrong room.

Now imagine the opposite. The control test has the right Silver chunk, and the model still says "covered" in 6 runs out of 10 (made up). Then the model is part of the problem. Try cheap fixes before replacing it.

• Make the task smaller. Do the date maths in code and tell the model "the customer is 14 months into a 24-month waiting period".
• Put the most important chunk first. A 2023 study of question answering over many documents found the models it tested often did best when the needed text sat near the start or end of the input.
• Ask for a structured answer with a field for the source, so your code can check it.
• Move hard rules, like waiting periods, into code instead of trusting the model with them.
• Only then compare other models on the same test cases, and choose on results, not reputation.

The two main types are factuality errors, which are wrong about the real world, and faithfulness errors, which do not match the model's input or instructions, or contradict themselves. Knowing which you have tells you where to look.

A 2023 survey of hallucination research, revised in 2024, splits these two families into five types. The table gives each one with an example from the insurance bot.

The customer's case fits nowhere neatly. The model mostly repeated the text it got. Missing that the chunks were about Gold while the prompt said Silver was a small faithfulness slip.

From her side it is still a made-up promise. The main fix still belongs in retrieval.

Ask the model to name each chunk it used, with a short exact quote, and have your code check every citation before the reply is shown. Grounding keeps the model to sources you chose; the check confirms it stayed there.

Published guidance points the same way. OWASP, a nonprofit foundation that publishes software security guidance, keeps a 2025 list of top risks for LLM applications. It counts misinformation as one risk. Its suggested fixes include retrieving verified information from trusted sources and automatically validating key outputs.

Here is that check in code. It rejects a citation whose chunk was never retrieved or is about another plan, whose quote is under three words, or whose chunk does not contain the exact quoted words. The retrieved list still holds a Gold chunk, as it would without the plan filter, so you can see each check fire.

CHUNKS = {
    "silver-waiting": ("Silver", "Silver plan waiting period for joint replacement is 24 months"),
    "gold-knee": ("Gold", "Gold plan covers knee replacement surgery up to Rs 3 lakh"),
}

def citation_problems(reply, retrieved, plan):
    if not reply["citations"]:
        return ["no source cited"]
    problems = []
    for cite in reply["citations"]:
        cid, quote = cite["id"], cite["quote"]
        if cid not in retrieved:
            problems.append(f"{cid} was never retrieved")
        elif CHUNKS[cid][0] not in (plan, "All plans"):
            problems.append(f"{cid} is about another plan")
        elif len(quote.split()) < 3:
            problems.append(f"quote too short to check in {cid}")
        elif quote not in CHUNKS[cid][1]:
            problems.append(f"quote not found in {cid}")
    return problems

retrieved = ["silver-waiting", "gold-knee"]
replies = {
    "good": {"citations": [{"id": "silver-waiting", "quote": "is 24 months"}]},
    "wrong plan": {"citations": [{"id": "gold-knee", "quote": "up to Rs 3 lakh"}]},
    "made-up quote": {"citations": [{"id": "silver-waiting", "quote": "is 12 months"}]},
    "one-word quote": {"citations": [{"id": "silver-waiting", "quote": "is"}]},
    "no source": {"citations": []},
}
for name, reply in replies.items():
    print(f"{name}: {citation_problems(reply, retrieved, 'Silver') or 'show it'}")

# Expected output:
# good: show it
# wrong plan: ['gold-knee is about another plan']
# made-up quote: ['quote not found in silver-waiting']
# one-word quote: ['quote too short to check in silver-waiting']
# no source: ['no source cited']

What just happened? Only the first reply passes. The Gold quote is real text, but from the wrong plan's brochure. The 12-month quote looks plausible, but those words are not in the chunk. The one-word quote "is" is so common that it proves nothing.

So keep the hard rules in code as well. The citation check catches the wrong brochure; the waiting-period sum in code catches a right brochure read wrongly.

Rerun enough times that a bug at the rate you care about would almost surely show up. Five clean runs feel convincing, but they prove little against a bug that appears 1 time in 10.

The arithmetic is short. If a bug appears in a fraction p of runs, the chance that n runs all pass anyway is (1 - p) multiplied by itself n times. The script turns that into run counts.

import math

def runs_to_catch(fail_rate, confidence=0.95):
    # smallest n where (1 - fail_rate) ** n <= 1 - confidence
    return math.ceil(math.log(1 - confidence) / math.log(1 - fail_rate))

for rate in (0.5, 0.1, 0.02):
    print(f"bug in {rate:.0%} of runs: rerun {runs_to_catch(rate)} times")

print(f"chance a 10% bug passes 5 runs: {0.9 ** 5:.0%}")
n = 30
print(f"0 failures in {n} runs: rate could still be up to "
      f"{1 - 0.05 ** (1 / n):.1%} (rule of three says {3 / n:.0%})")

# Expected output:
# bug in 50% of runs: rerun 5 times
# bug in 10% of runs: rerun 29 times
# bug in 2% of runs: rerun 149 times
# chance a 10% bug passes 5 runs: 59%
# 0 failures in 30 runs: rate could still be up to 9.5% (rule of three says 10%)

What just happened? A bug in 10% of runs passes 5 runs in a row 59% of the time. To be 95% sure of seeing it at least once, you need 29 runs. A 1-in-50 bug needs 149.

The reverse question has a handy shortcut from medical statistics, the rule of three. If you see zero failures in n runs, the true failure rate could still be up to about 3 divided by n, at 95% confidence. The shortcut is close from about 30 runs up; at 5 runs the exact limit is 45%, not 60%.

So 30 clean runs still allow a bug in up to about 10% of runs; the exact figure is 9.5%. For a wrong yes that can cost a customer Rs 2.4 lakh, move the rule into code instead.

The maths assumes every run is a fresh, independent try. If your setup can hand back a stored reply for a repeated prompt, those repeats are not new evidence.

Fix the bug in the layer where it started, then add the failing chat to your regression set so every later change is tested against it. A fix in the wrong layer can hide the bug for one example and leave it alive for others.

The tempting wrong fix here was a louder prompt, such as a capital-letter order never to say covered unless certain. That would not stop the search from pulling Gold chunks for the next Silver customer.

The engineer makes three fixes. Retrieval, the root cause: the search now filters chunks by the customer's plan. Prompt, the contributing cause: the model may say "not sure" and must cite chunks with quotes.

Code, a new guard: the bot works out the months left in the waiting period itself, and the citation check runs before every reply.

To check the retrieval fix, run the search with and without the plan filter. This toy search scores chunks by shared words. Real search is smarter, but no scorer can know the customer's plan unless you pass it in.

CHUNKS = [  # (id, plan, text): six brochure chunks
    ("gold-knee", "Gold", "Gold plan covers knee replacement surgery up to Rs 3 lakh"),
    ("gold-waiting", "Gold", "Gold plan waiting period for knee replacement is 12 months"),
    ("claims-howto", "All plans", "How to file a claim when surgery is covered"),
    ("gold-limits", "Gold", "Gold plan limits for knee surgery and hospital rooms"),
    ("silver-rooms", "Silver", "Silver plan room rent limit is Rs 5000 per day"),
    ("silver-waiting", "Silver", "Silver plan waiting period for joint replacement is 24 months"),
]
STOP = {"is", "my", "a", "for", "the", "to", "at", "up", "and", "when"}

def words(text):
    return {w.strip("?.,").lower() for w in text.split()} - STOP

def search(question, k=4, plan=None):
    pool = [c for c in CHUNKS if plan is None or c[1] in (plan, "All plans")]
    ranked = sorted(pool, key=lambda c: len(words(question) & words(c[2])),
                    reverse=True)
    return [c[0] for c in ranked[:k]]

q = "Is my knee replacement surgery covered?"
print("Before:", search(q))
print("After: ", search(q, plan="Silver"))

# Expected output:
# Before: ['gold-knee', 'gold-waiting', 'claims-howto', 'gold-limits']
# After:  ['claims-howto', 'silver-waiting', 'silver-rooms']

What just happened? Without the filter, the top four match the trace: three Gold chunks and a general one. With it, silver-waiting makes the cut in second place, so the rule the model needed now reaches the prompt. Second is not first, so the "most important chunk first" advice still applies.

The table maps common symptoms to the layer you should fix. Use it as a first guess, then confirm with a controlled change.

Adding the case to a regression set

Every real failure becomes a test case. The case below stores a test policy, not the customer's real data. It also holds the question, the right answer, the chunk that must be cited and a note on its origin.

CASES = [
    {"id": "silver-knee-14-months", "policy_id": "TEST-SIL-0001",
     "question": "Is my knee replacement surgery covered?",
     "expected_covered": False, "must_cite": "silver-waiting",
     "note": "Bot quoted the Gold brochure to a Silver customer"},
]

def run_regression(cases, ask_bot, runs=5):
    failed = []
    for case in cases:
        for run in range(1, runs + 1):
            reply = ask_bot(case["policy_id"], case["question"])
            if reply["covered"] != case["expected_covered"]:
                failed.append((case["id"], run, "wrong coverage answer"))
            elif case["must_cite"] not in reply["cited_ids"]:
                failed.append((case["id"], run, "right answer, wrong source"))
    return failed

# Two stand-ins so the script runs; use your real bot here.
def bot_before_fix(policy_id, question):
    return {"covered": True, "cited_ids": ["gold-knee"]}

def bot_after_fix(policy_id, question):
    return {"covered": False, "cited_ids": ["silver-waiting"]}

print(run_regression(CASES, bot_before_fix, runs=2))
print(run_regression(CASES, bot_after_fix))

# Expected output:
# [('silver-knee-14-months', 1, 'wrong coverage answer'), ('silver-knee-14-months', 2, 'wrong coverage answer')]
# []

What just happened? The old bot fails both runs on coverage. The fixed bot passes all five runs, so the list is empty. The must_cite check also catches a right answer given for the wrong reason.

Run the set before every change to the prompt, the search, the chunks or the model. Pick the run count with the arithmetic above: 5 runs per case is a quick smoke test (a fast check that nothing is badly broken), not proof.

Give a method, not a guess: reproduce, walk the layers in order, prove the cause with one change, fix that layer, and add a regression case. Interviewers listen for whether you blame the model too early.

1. 1

   Describe the symptom exactly

   Say what was wrong and what it cost: the bot gave a Silver customer a Gold plan rule, and she paid Rs 2.4 lakh herself.

2. 2

   Reproduce from the trace

   Replay the saved prompt many times and count failures, so you know the bug is real and how often it happens.

3. 3

   Walk the layers in order

   Inputs and retrieved context, then prompt, then tool results, then parsing, then the model.

4. 4

   Prove the cause

   Run a control test with perfect context. If the answer becomes correct, the model is not the root cause.

5. 5

   Fix the right layer and guard it

   Filter retrieval by plan, move hard rules into code, check citations, and add the case to the regression set.