RAG Evaluation Metrics: Recall@k, MRR and Context Precision

You learn that answers are wrong, but not why: search may have missed the right text, or the model may have misused it. RAG evaluation metrics split the score so you fix the half that actually broke.

Picture a common setup. A small tax-advice firm runs a help assistant that answers salaried people's income-tax questions in English or Hinglish (Hindi and English mixed). It answers from the firm's 600 help notes. The setup is made up, but the problem is real, and nothing here is tax advice.

One evening, the chartered accountant who wrote most of the notes read 50 recent answers and marked 12 of them wrong. One taxpayer had asked whether rent paid to their mother can be claimed. The assistant said yes, then added a yearly limit that appears in no note at all.

The engineer who built the assistant wanted to rewrite the prompt, the instructions sent to the model with every question. The accountant asked a sharper question. Did the assistant miss the right note, or find it and then get it wrong?

RAG evaluation metrics are numbers that each score one part of a RAG system, measured on questions an expert has already answered. Retrieval metrics score the search, and generation metrics score the written answer.

RAG, short for retrieval-augmented generation, works in two steps. First a search step picks the few passages most likely to help. Then the model, the program that reads text and writes text back, writes the answer from those passages.

So every wrong answer has two suspects. This guide gives each suspect its own metrics, works every metric through the rent question with real arithmetic, and ends with a way to name the guilty half.

You need four: chunk, top-k, labelled question set and judge model. Each gets a plain meaning and an example from the tax assistant.

Chunk. A chunk is a small piece of a document, often a few paragraphs, stored so it can be searched on its own. The firm's 600 help notes are split into about 4,000 chunks, each with an id like c12.

Top-k. The retriever, the search part of a RAG system, returns chunks ranked from most to least likely to help. Top-k means the first k results in that list. The assistant sends the top 5 chunks to the model, so k is 5.

Labelled question set. This is a list of test questions where an expert has marked, in advance, the chunk ids that hold each answer. The expert also writes a short reference answer. People also call it a golden set.

Judge model. A judge model is a model you ask to grade another model's work, such as whether one sentence of an answer is backed by chunk c12. It is fast but can be wrong, so a person checks a sample of its grades.

Measure them separately because they fail for different reasons and need different fixes. One overall score tells you something is wrong, but not where.

Think of an open-book exam. Opening the book at the right page is retrieval. Writing the answer from that page is generation. A student can lose marks by opening the wrong chapter, or by opening the right one and copying the wrong line.

A teacher who sees only the final mark cannot tell which happened. If the note about rent paid to parents never reaches the model, even a perfect model cannot answer well. If it arrives and the model invents a limit, writing failed.

Where the picture breaks: a model, like a student, can also answer from memory instead of the page. That is why faithfulness checks the answer against the retrieved chunks, not just against the right answer.

The diagram below shows the assistant's pipeline, the chain of steps from question to answer. Notice where each check plugs in: the retrieval check needs only the ranked chunk ids, and the generation check needs the answer and its chunks.

This split also saves money. Retrieval metrics are plain arithmetic on chunk ids, so they need no model calls beyond the search itself. Generation metrics need at least one judge-model call per answer, so they cost more and run slower.

You build it by collecting real questions, then having an expert mark which chunks answer each one and write a short reference answer. Every metric in this guide is only as good as these labels.

The engineer and the accountant built the first set over two weeks, in six steps.

1. 1

   Collect real questions

   Pull 300 questions from the assistant's logs, after removing names, PAN (tax ID) numbers and phone numbers. Real wording matters, because taxpayers rarely use the words the notes use.

2. 2

   Remove repeats and balance topics

   Keep 200, spread across salary, rent, capital gains, refunds and tax notices, so one popular topic cannot swamp the score.

3. 3

   Mark the relevant chunks

   For each question, the accountant lists every chunk that holds part of the answer. The rent question got two: c12 on rent paid to family, and c47 on the proof you need.

4. 4

   Write a reference answer

   The accountant writes a correct answer in two or three sentences, for the generation metrics that need one.

5. 5

   Add questions with no answer

   Add 20 questions the notes cannot answer, such as questions about another country's taxes. The right reply there is to say so, not to guess.

6. 6

   Freeze it and version it

   Save all 220 questions as a file with a version number, and keep 30 more hidden for a final check after tuning.

A few habits keep the set honest:

• Keep real phrasing, typos and Hinglish. A set written in clean English by the engineer will flatter the retriever.
• Have a second person label a sample. If they often disagree with the accountant, the labels are not ready.
• Update labels when the notes change, or a deleted chunk makes the retriever look worse than it is.
• Turn every real failure the accountant spots into a new test question.

Recall@k is the share of the labelled chunks that appear in the top k results. Precision@k is the share of the top k results that are labelled chunks. In RAG, recall@k usually matters most, because the model only ever sees the top k.

Take the rent question. The accountant labelled two relevant chunks, c12 and c47. The retriever's top 5, in order, was c08, c12, c33, c91 and c05. Chunk c12 is second, and c47 is missing.

Recall@5 is 1 found out of 2 relevant, which is 0.5. Half of what the model needed never arrived. Precision@5 is 1 relevant out of 5 returned, which is 0.2. Four of the five chunks were extra text the model had to read past.

Averaged across the 200 answerable questions, recall@5 was 0.62. So on average, about 38 percent of each question's labelled chunks never reached the model.

Some teams use a simpler version called hit rate@k: did at least one relevant chunk appear in the top k? It would give the rent question full marks, and hide the missing chunk c47.

MRR, or mean reciprocal rank, scores how high the first relevant chunk appears in the list. For each question, take 1 divided by the position of the first relevant chunk, then average across all questions.

Reciprocal just means 1 divided by a number. First place scores 1, second place 0.5, third place about 0.33 and fourth place 0.25. If no relevant chunk appears in the top k, the question scores 0.

Here are four questions from the set and their scores:

• Rent paid to mother: first relevant chunk at position 2, so 1 divided by 2, or 0.5.
• Fee for filing late: first relevant chunk at position 1, so 1.
• Refund not received: no relevant chunk in the top 5, so 0.
• Selling gold jewellery: first relevant chunk at position 4, so 0.25.

Add them up to get 1.75, then divide by 4 questions: MRR is 0.4375, about 0.44. An MRR near 1 means the right chunk is almost always first.

MRR only looks at the first relevant chunk, so for the rent question it ignores the missing c47 completely. That is fine when one chunk is enough to answer. When answers need two or three chunks together, report recall@k next to it.

Here is the arithmetic in Python. The list retrieved holds chunk ids in the order the retriever ranked them. The set relevant holds the ids the accountant labelled for that question. Run it with any Python 3 and compare the printed numbers with the comments.

def recall_at_k(retrieved, relevant, k):
    found = [c for c in retrieved[:k] if c in relevant]
    return len(found) / len(relevant)

def precision_at_k(retrieved, relevant, k):
    found = [c for c in retrieved[:k] if c in relevant]
    return len(found) / k

def reciprocal_rank(retrieved, relevant, k):
    for position, chunk_id in enumerate(retrieved[:k], start=1):
        if chunk_id in relevant:
            return 1 / position
    return 0.0

retrieved = ["c08", "c12", "c33", "c91", "c05"]
relevant = {"c12", "c47"}
print(recall_at_k(retrieved, relevant, 5))      # 0.5
print(precision_at_k(retrieved, relevant, 5))   # 0.2
print(reciprocal_rank(retrieved, relevant, 5))  # 0.5

# MRR over the four questions above: first hits at 2, 1, none, 4
first_hits = [2, 1, None, 4]
scores = [1 / p if p else 0.0 for p in first_hits]
print(sum(scores) / len(scores))                # 0.4375

What just happened? The slice retrieved[:k] cuts the list to the chunks the model actually saw. Recall divides by the number of labelled chunks, precision divides by k, and MRR is the average of the reciprocal ranks.

Leave the 20 no-answer questions out of recall. They have no relevant chunks, so recall_at_k would divide by zero. Score them separately, by whether the assistant said it could not answer.

nDCG, short for normalised discounted cumulative gain, scores the order of the whole ranked list. It rewards relevant chunks near the top more than relevant chunks lower down, and a perfect order scores 1.

It works in three steps. Give each position a weight that shrinks down the list. Add up the weights of the positions holding relevant chunks: that is the DCG. Then divide by the best total possible, with all labelled chunks at the top.

The usual weight is 1 divided by log base 2 of (position plus one). Positions 1 to 5 count 1, 0.63, 0.5, 0.43 and 0.39.

Now compare two retrievers the engineer tried on the rent question. Both found c12 and c47 in the top 5, so both have recall@5 of 1.0. Retriever A put them in positions 1 and 5. Retriever B put them in positions 4 and 5.

Retriever A gets 1 plus 0.39, which is 1.39. Retriever B gets 0.43 plus 0.39, which is 0.82. The best order puts both chunks first and second, worth 1 plus 0.63, or 1.63. So nDCG@5 is about 0.85 for A and about 0.50 for B.

Recall@5 called the two retrievers equal, but nDCG shows that A is much better. This matters if the team ever cuts k from 5 to 3: retriever A would still deliver one needed chunk, and retriever B would deliver none.

nDCG can also use graded labels: the accountant could mark a chunk 2 if it fully answers the question, and 1 if it only helps. A 2021 benchmark paper comparing search systems across 18 public datasets uses nDCG@10 as its single main score.

Its authors chose nDCG because it handles both yes-or-no labels and graded labels. They note that MRR cannot handle graded labels.

Four common ones are faithfulness, answer relevance, context precision and context recall. The first two grade the answer itself. The last two grade the chunks the model was given, judged against the reference answer.

Faithfulness and answer relevance grade the answer

Faithfulness asks whether every claim in the answer is backed by the retrieved chunks. Some evaluation tools score a similar check under the name groundedness. A 2023 paper on automated RAG evaluation scores it simply.

Split the answer into short claims, ask a judge whether each one can be inferred from the chunks, and divide supported claims by total claims.

Example: the rent answer made four claims. Three appear in chunk c12, but the yearly limit appears nowhere, so faithfulness is 3 out of 4, or 0.75. An invented claim like that is called a hallucination.

Answer relevance asks whether the answer addresses the question asked. A faithful answer can still miss the point. An early version of the assistant, asked about rent paid to a mother, explained rent receipts in general and never said yes or no.

The same 2023 paper measures relevance indirectly. A model writes a few questions that the answer would fit, and the score is how close those questions are in meaning to the real one.

Context precision and context recall grade the chunks

Context precision asks how much of the retrieved text was useful. A 2024 paper on diagnosing RAG systems counts a chunk as relevant if it holds at least one fact from the reference answer. Its score is relevant chunks divided by retrieved chunks.

For the rent question, only c12 holds a fact from the reference answer, so context precision is 1 out of 5, or 0.2. Other tools compute this name differently, which the engineer sections cover.

Context recall asks whether the retrieved chunks hold every fact the correct answer needs. The 2024 paper calls it claim recall. A judge splits the reference answer into facts and checks each against the chunks. The reference answer has 4 facts and only 3 appear, so context recall is 0.75.

The missing fact is the proof rule in c47, which never arrived. These two sound like precision@k and recall@k, but here a judge model reads the text, so you can use them before you have chunk labels.

The map below shows what each metric compares: each arrow joins the two things one metric compares. Arrows that touch the reference answer need the accountant's work in advance. Faithfulness and answer relevance do not.

Answer correctness, the last arrow, compares the answer with the reference answer. It tells you an answer is wrong, but not why.

Here is faithfulness in Python, plus a function that points at the failing half. It uses two helpers you write yourself around your judge model. The helper split_into_claims breaks an answer into short claims, and judge_supports returns True or False for one claim.

def faithfulness(answer, chunks):
    claims = split_into_claims(answer)
    if not claims:
        return 1.0          # a refusal has nothing to check; some tools return NaN
    supported = [c for c in claims if judge_supports(c, chunks)]
    return len(supported) / len(claims)

def which_half_failed(item, retrieved_ids, chunks, answer):
    if not item.relevant:
        return "no-answer question"  # check whether it refused
    if recall_at_k(retrieved_ids, item.relevant, 5) < 1.0:
        return "retrieval"          # needed text never reached the model
    if faithfulness(answer, chunks) < 0.8:
        return "generation"         # the model added claims of its own
    return "check by hand"          # relevance, the label or the note itself

An answer with no claims, such as a polite refusal, scores 1.0 here because nothing in it is unsupported. The team runs which_half_failed only on answers the accountant marked wrong. A no-answer question returns early, since recall needs labelled chunks, so you check whether the assistant refused. The 0.8 faithfulness line is the team's own choice, not a standard.

The rent answer fails both ways, but the function reports retrieval because it checks that first. The team fixes search, reruns the question, then checks whether the invented limit remains.

The table below collects the main metrics, scored on the rent question. Its nDCG@5 of about 0.39 is one relevant chunk in position 2, worth 0.63, divided by the best possible 1.63.

Check retrieval first: if the needed chunks were not in the top k, it is a retrieval failure, whatever the model wrote. Only when the right chunks arrived do you look at faithfulness and answer relevance.

The accountant first labelled the chunks each of the 12 wrong answers needed. Then the team walked them through the chart below. Start at the top with one wrong answer and follow the arrows that fit it.

Here is what they found. Eight of the 12 failed at the first question, because a needed chunk was never in the top 5. Three failed at the second: the chunks were right, but the model added a fact. One passed every check, and its note was simply out of date.

So the prompt change would have touched only 3 of the 12 failures. Most of the problem was search: many missed chunks came from notes whose tables had been split down the middle.

Each half has its own usual fixes, and its own metric to watch:

• Needed chunks missing: change how notes are split into chunks, add keyword search, or rewrite the question before searching. Watch recall@k.
• Right chunks, poor order: add a reranker, a second model that re-sorts the top results more carefully. Watch MRR and nDCG.
• Claims not in the chunks: tell the model to answer only from the chunks, and to say when they are not enough. Watch faithfulness.
• Answers that wander: ask for a direct yes or no first, then the reason. Watch answer relevance.

The engineer fixed how tables were split, and average recall@5 rose from 0.62 to 0.81 on the same frozen set (made up). Because the questions and labels did not change, the gain came from the fix.

Two tools can disagree because they build the best possible order from different labels, or turn grades into gains with different formulas. Both choices are invisible in a dashboard that shows only the final number.

The ideal order must include chunks that were never found

nDCG divides by the score of the best possible order. That best order must include every labelled chunk, including c47, which the retriever never returned.

Some tools only see the chunks you pass in, which are the five that came back. They cannot know c47 exists, so their best order holds only c12. The rent question then scores 0.63 instead of 0.39, and a missed chunk stops costing anything.

Graded labels can use two different gain formulas

With yes-or-no labels, a relevant chunk adds a gain of 1 and both formulas agree. With grades, some tools use the grade itself as the gain. The textbook formula uses 2 to the power of the grade, minus 1, which rewards the best chunks more.

Run this check yourself with any Python 3. It prints the rent question, both retrievers, the trap and both gain formulas, and each comment shows the number you should see.

import math

def dcg(gains):
    return sum(g / math.log2(pos + 1) for pos, g in enumerate(gains, start=1))

def ndcg_at_k(retrieved, labels, k):
    gains = [labels.get(c, 0) for c in retrieved[:k]]
    ideal = sorted(labels.values(), reverse=True)[:k]
    return dcg(gains) / dcg(ideal)

labels = {"c12": 1, "c47": 1}           # every labelled chunk, found or not
live = ["c08", "c12", "c33", "c91", "c05"]
print(round(ndcg_at_k(live, labels, 5), 3))                                  # 0.387
print(round(ndcg_at_k(["c12", "c08", "c33", "c91", "c47"], labels, 5), 3))  # 0.85
print(round(ndcg_at_k(["c08", "c33", "c91", "c12", "c47"], labels, 5), 3))  # 0.501

# The trap: labels limited to the five chunks that came back
seen_only = {c: g for c, g in labels.items() if c in live}
print(round(ndcg_at_k(live, seen_only, 5), 3))                               # 0.631

# Graded labels: c12 fully answers (2), c47 only helps (1)
graded = {"c12": 2, "c47": 1}
print(round(ndcg_at_k(live, graded, 5), 3))                                  # 0.48
power = {c: 2 ** g - 1 for c, g in graded.items()}
print(round(ndcg_at_k(live, power, 5), 3))                                   # 0.521

The trap line prints 0.631 for the same ranking that honestly scores 0.387. With graded labels, the two formulas give 0.48 and 0.521. Write the label scope and the gain formula next to every nDCG number you report.

Judge-based metric names are not standardised, so two tools can compute different maths under one name. Check the definition in your tool's source or docs before you compare numbers.

Context precision is the clearest case. The 2024 paper above scores the share of retrieved chunks that hold a reference fact, which gives the rent question 0.2.

One popular open-source RAG evaluation library uses a rank-weighted average instead, in its latest release as of 19 October 2026. In its default version, a judge marks each chunk useful or not against the reference answer. The score then rewards useful chunks near the top.

For each useful chunk, take the precision at its position, then average over the useful chunks. With c12 useful at position 2, the score is 0.5. If c12 had come first, it would be 1.0.

The same retrieval therefore scores 0.2 in one tool and 0.5 in another. The 2023 paper does not use the name at all: its context relevance counts the share of retrieved sentences needed to answer.

Refusals differ too. The faithfulness function above returns 1.0 for an answer with no claims. The same library returns NaN when it finds no claims, short for not a number, so refusals drop out of some averages and break others.

Details like these, some found only in the source code, can change between releases, so they hold as of 19 October 2026. Pin the tool version, and store each judge's prompt with the scores it produced.

Explain it as two scorecards, one for search and one for writing, built on a labelled question set. Interviewers want to hear how you find the broken half, not a list of metric names.

1. 1

   Split the pipeline

   Say you score retrieval and generation separately, because they fail for different reasons and need different fixes.

2. 2

   Describe the question set

   Real user questions, expert chunk labels and reference answers, some no-answer questions, all frozen and versioned.

3. 3

   Name the retrieval metrics

   Recall@k, with k equal to the number of chunks the model sees, plus MRR or nDCG for the order.

4. 4

   Name the generation metrics

   Faithfulness and answer relevance from a judge model, checked against a sample of expert grades.

5. 5

   Walk through one failure

   Take one wrong answer, like the rent question, through the checks. Name the broken half and your fix.

6. 6

   Keep it running

   Run retrieval checks on every change and generation checks nightly, and add every real failure to the set.