LLM Evaluation Metrics and Test Frameworks Explained

A small change to a prompt or model can quietly break answers that used to work, and a handful of questions typed by hand will not show it. LLM evaluation metrics fix this by scoring your app on the same fixed set of test questions after every change.

Picture a common setup. An online pharmacy runs a chat assistant for two kinds of questions: where an order is, and whether a medicine is in stock. The setup is made up, but the problem is real. The assistant must never give medical advice, so questions about doses or side effects go to the pharmacist.

One Tuesday, the engineer who looks after the assistant changed the prompt (the written instructions sent to the model with every question) to make replies sound friendlier. The engineer typed five questions into the chat box, liked the answers and shipped the change.

Here is what happened next, with made-up numbers. The assistant handles about 1,800 chats a day. Over the next week it told some customers that metformin 500 mg was in stock when the shelf was empty. Then 23 orders had to be cancelled, each costing around Rs 120 in refund work and courier charges.

Worse, the pharmacist found one chat where the assistant told a customer how many tablets to take. It was caught only because the pharmacist reads 20 random chats by hand every Friday. Nobody had planned to test that kind of question, so nobody saw that the friendlier prompt had broken it.

Five questions typed by hand is a demo, not a test. This guide replaces it with a fixed set of questions, clear scores and a check on every change, built step by step on this pharmacy assistant.

You need six words: evaluation, golden test set, reference answer, metric, grader and regression. First, LLM is short for large language model, the kind of AI model behind the assistant. It reads text and writes text back.

Evaluation. An evaluation, often shortened to eval, is a repeatable test of how well your AI system answers. Example: run 300 saved questions through the assistant and count how many answers are good.

Golden test set. This is the fixed list of test questions, each with the answer or behaviour you expect. Example: "Is paracetamol 650 in stock?" paired with "Yes, 140 strips available", based on a copy of the stock sheet saved with the test.

Reference answer. The reference is the correct answer for a test question, written in advance by someone who knows. Example: the pharmacist writes "Hand over to the pharmacist" as the reference for a question about mixing a medicine with a blood pressure tablet.

Metric. A metric is a rule that turns many answers into one number. Example: the share of order questions where the assistant gave the right order status, which might come out as 0.94.

Grader. A grader is whatever decides if one answer passes: a few lines of Python, a person, or another AI model used as a judge. Example: a Python check that the reply contains no tablet count.

Regression. A regression is something that used to work and now does not. Example: the friendlier prompt let dose questions slip past the handover rule. Regression tests exist to catch exactly this.

The main LLM evaluation metrics are exact match, precision and recall, rule checks, faithfulness, rubric scores from an AI judge, and human ratings. They fall into two families: reference-based and reference-free.

Reference-based or reference-free?

A reference-based metric compares the answer with a reference answer written in advance. Example: the reference says the order status is "shipped", and you check whether the assistant said "shipped".

A reference-free metric judges the answer without a written correct answer. It checks the answer against rules, or against the documents the assistant was given. Example: checking that the reply contains no tablet count needs no reference.

Reference-based metrics are more exact but need an expert to write answers. Reference-free metrics work on any question, even new ones, but say less about full correctness. Most teams use both.

When is exact match enough?

Exact match means the answer counts as correct only if it equals the reference, usually after small clean-ups like lowercasing and trimming spaces. It works when the answer is a label, a number or a status. It fails for normal sentences: "It has shipped" and "Your parcel is on its way" mean the same thing but do not match.

So the team makes the assistant fill a small structured field (a fixed slot with only a few allowed values) before it writes its friendly reply. Example, written in JSON, a simple text format for data: {"intent": "order_status", "status": "shipped"}. Exact match grades the field. Other metrics grade the sentence.

What are precision and recall for an LLM?

Precision and recall measure how well the assistant sorts questions into groups, scored for the one group you care about most. Here that group is "pharmacist": questions the assistant must hand over to the pharmacist.

Recall asks: of all the questions that truly needed the pharmacist, how many did the assistant hand over? Precision asks: of all the questions it handed over, how many truly needed the pharmacist? F1 combines the two into one number that drops sharply if either is low.

Using made-up numbers from an early test run, the test set has 40 pharmacist questions. The assistant handed over 45 questions in total, and 36 of them were real pharmacist questions. Recall is 36 out of 40, which is 0.90. Precision is 36 out of 45, which is 0.80.

Which matters more? The 4 missed questions are customers who may have got medical advice from a chatbot. The 9 extra handovers only cost the pharmacist a few minutes. So for this group, recall matters far more, and one overall score would hide that.

Here is the scoring in Python. Each result holds the label a person expected and the label the assistant chose. The made-up list at the bottom rebuilds the run above, so you can check the numbers yourself.

from collections import namedtuple

Result = namedtuple("Result", "expected predicted")


def score_routing(results, group="pharmacist"):
    exact = sum(r.predicted == r.expected for r in results) / len(results)
    sent = [r for r in results if r.predicted == group]
    needed = [r for r in results if r.expected == group]
    caught = [r for r in needed if r.predicted == group]
    precision = len(caught) / len(sent) if sent else 0.0
    recall = len(caught) / len(needed) if needed else 0.0
    f1 = 2 * precision * recall / (precision + recall) if caught else 0.0
    return {"exact_match": round(exact, 3), "precision": round(precision, 3),
            "recall": round(recall, 3), "f1": round(f1, 3)}


# Made-up run of 300 questions: 36 pharmacist questions caught, 4 missed,
# 9 stock questions wrongly handed over, 251 routed correctly elsewhere
results = ([Result("pharmacist", "pharmacist")] * 36
           + [Result("pharmacist", "stock")] * 4
           + [Result("stock", "pharmacist")] * 9
           + [Result("stock", "stock")] * 251)
print(score_routing(results))

Run it with python3 and it prints exact_match 0.957, precision 0.8, recall 0.9 and f1 0.847. The if checks stop a division by zero when a group is empty.

What just happened? Exact match is 287 of 300, which looks excellent. Yet recall shows that 4 risky questions slipped past. That is why you score the risky group on its own.

The table compares all six metrics. Pick the cheapest one that answers your question.

You build a golden test set by collecting real questions, cleaning them, adding hard cases, and having an expert write the expected answer for each. Then you freeze it as a versioned file that every test run uses.

Think of the monthly stock check in a kirana shop (a small neighbourhood grocery). The owner does not trust memory: each month they go down the same printed list of items and count each one. Each item on the list is a test question, the counting rules are your metrics, and each month's tally is the score of one run.

Where the picture breaks: a stock check only counts what is on the shelf. A golden test set also needs an expert's judgement on what a good answer is, and it must grow as customers find new ways to break the assistant.

The diagram shows where the test questions come from.

Here are the steps the team followed for the first version.

1. 1

   Write down what good means

   For each question type, write one sentence. Order questions: the right status and date. Stock questions: the right yes or no, from the saved stock sheet. Medical questions: hand over to the pharmacist, with no advice at all.

2. 2

   Collect real questions

   Export recent chats and strip out names, phone numbers and anything about a person's health history. Real wording beats invented wording, because customers type in ways you would never guess.

3. 3

   Choose a balanced mix

   Pick questions so that rare but risky types are well represented, not just the common ones. The made-up mix: 120 order questions, 110 stock questions, 40 medical questions that must be handed over, and 30 others.

4. 4

   Add hard cases by hand

   Write questions that are likely to break things. Use misspelled medicine names and dose questions hidden inside order questions. Add Hinglish (Hindi typed in English letters) such as "metformin hai kya?".

5. 5

   Write references and freeze the file

   The pharmacist writes the expected answer or behaviour for each question. The engineer saves the set as a versioned file, so last month's score and this month's score are measured on the same questions.

The test set keeps growing. Every time a customer finds a new way to break the assistant, the engineer cleans that chat and adds it as a new question. The dose chat the pharmacist found on Friday was one of the first to go in.

You measure faithfulness by splitting the answer into short claims and checking whether each claim is supported by the documents the system retrieved. The score is the share of claims that are supported.

First, the word RAG. Retrieval-augmented generation (RAG) means your program first searches your own data, then gives the matching pieces to the model along with the question. Here, the program looks up the medicine in today's stock sheet and pastes that row into the prompt. The pasted row is called the context.

Faithfulness asks one thing: did the answer stick to the context? A 2023 paper by Es et al. defines it this way. A model splits the answer into short statements and checks each against the context. The score is supported statements divided by all statements.

Here is a made-up example. The context says: "Metformin 500 mg: 0 strips. Restock expected Thursday." The assistant replied: "Yes, metformin 500 mg is in stock, and we can deliver it tomorrow. More is arriving Thursday."

That reply has three claims. "In stock" is not supported, because the sheet says 0. "Deliver tomorrow" is not supported, because the sheet says nothing about delivery. "Arriving Thursday" is supported. So faithfulness is 1 out of 3, about 0.33, the kind of bug behind the 23 cancelled orders.

Follow the diagram from the question to the score.

In that paper both steps, splitting and checking, are done by a model, which is why the table counts at least two AI calls per answer. For short, structured context like a stock row, plain code can sometimes do it: if the sheet says 0 strips, the reply must not say "in stock".

Test retrieval separately too: did the right row come back for each stock question? Then low faithfulness points to the prompt or model, and wrong rows point to the search.

LLM-as-a-judge means you ask a second AI model to grade your assistant's answer against a written rubric, and it returns a verdict with a reason. You can trust it only after you have checked its verdicts against a human expert on a sample.

A rubric is a short list of things a good answer must do, each checked on its own. The team's rubric for stock answers has four points, and the judge gives a pass or fail on each.

• The stock status matches the context exactly.
• If the medicine is out of stock and the context has a restock date, the reply mentions it.
• The reply gives no dose, timing or side-effect advice.
• The reply is polite and under 80 words.

Narrow pass or fail points are easier to check than "rate this from 1 to 10". A 7 is hard to compare with a human's 7, but "mentions the restock date" is simply true or false.

The judge also writes its reason before its verdict. A 2023 study by Wang et al. asked the judge for evidence first, sampled three verdicts and averaged over both answer orders. Together, these raised one judge's accuracy against human labels from 52.7% to 62.5%.

What biases does an AI judge have?

Research has found repeatable biases. Position bias: a judge's verdict can flip when you swap the two answers. In a 2023 study by Zheng et al., all three judges leaned towards the first answer with the default prompt. When the prompt gave the two answers new labels, one judge leaned towards the second instead, and Wang et al. also found judges favouring different slots.

Verbosity bias: judges can prefer longer answers. Zheng et al. padded 23 answers by restating their list items without adding information. Two of the three judges preferred the padded answer 91.3% of the time; the third did 8.7% of the time.

Self-enhancement bias: a judge may rate its own model's answers more kindly. Zheng et al. saw signs of this but say their data cannot determine whether the bias is real.

Maths and reasoning: in the same study, a judge was misled by a wrong answer to a basic maths question it could solve correctly when asked on its own. Giving the judge a reference answer cut failures on 10 maths questions from 14 of 20 judgements to 3 of 20.

Each bias has a partial counter, and you should test it on your own judge. When comparing two answers, run the judge twice with the order swapped, and call it a tie if the verdicts disagree. Zheng et al. did exactly this.

For length, keep a word limit in the rubric, so an overlong reply fails that point. An instruction alone may not be enough: that study's default judge prompt already said length should not sway the verdict. The paper does not say which prompt its padding test used, so try padded copies of your own answers.

As a precaution against self-enhancement bias, which that study could not confirm, use a different model as the judge from the one that wrote the answers, where you can. Check numbers with plain code, or give the judge a reference answer.

The diagram shows how the team earns trust in the judge.

In the made-up first round, the judge agreed with the pharmacist on 52 of 60 answers (87%). Six of the 8 disagreements were about one point: the judge passed replies that said "take it after food". The pharmacist counts that as advice.

The engineer made the rubric spell it out, and the next round agreed on 58 of 60. Later, the engineer section shows why 52 of 60 is less impressive than it sounds.

When do you still need human review?

You need people wherever the standard itself is being set, or where a wrong answer can hurt someone. The pharmacist's time is scarce, so it goes only where machines are weakest.

• Writing reference answers for new question types before anything can be graded.
• Labelling a fresh sample every few weeks, so the judge's agreement is rechecked.
• Reading every answer in the medical-risk part of the test set, because a miss there is the costliest mistake.
• Reviewing disagreements: answers where the judge and the rule checks gave different verdicts.

Human graders are not perfect either. The engineer and the pharmacist once graded the same 30 answers and disagreed on 3. Those 3 showed where the written definition of good was still vague.

You slice results because one overall average can hide a group of questions where the assistant fails badly. A slice is a group of test questions that share something, such as question type or language, scored on its own.

Pause and predict. With made-up numbers, the friendlier prompt passed 91% of all 300 questions. Before reading on, guess whether any group scored below 80%.

Two did. The 60 Hinglish questions passed only 75%, and the 25 questions with misspelled medicine names passed only 68%. The average was propped up by easy English order questions.

The team tags every question with its slices, using five kinds.

• Question type: order status, stock, pharmacist handover, other.
• Language: English, Hindi, Hinglish.
• Difficulty: clean spelling or misspelled medicine names.
• Risk: anything near medical advice, which has a zero-tolerance rule.
• Source: real customer chats or hand-written hard cases.

Slices also tell you where to look. If only the misspelled-name slice drops, the problem is probably in the stock search, not in the prompt.

A practical start is 50 to 100 well-chosen examples, growing towards a few hundred as you learn where the assistant fails. That is a working start, not a statistical rule. The real answer depends on how small a drop you need to catch.

For scale, a 2024 paper on eval statistics by Miller works through an example. To catch a 3-point difference between two models 80% of the time, with a 5% false-alarm rate, it needed about 1,000 independent questions. That example compares the two models question by question, and under its assumptions an unpaired comparison would need more.

Every test set is a sample, and samples are noisy. A margin of error is the plus-or-minus range around your score where the true pass rate very likely sits. For a pass rate p on n questions, the usual 95% margin is 1.96 x square root of (p x (1 - p) / n).

This formula assumes a bell-shaped spread of scores, so it gets unreliable when p is close to 0 or 1, or when a slice has only a few dozen questions. The script below works out margins for three cases, then compares two prompts question by question.

import math


def margin(p, n):
    return 1.96 * math.sqrt(p * (1 - p) / n)


for p, n in [(0.90, 100), (0.90, 400), (0.75, 60)]:
    print(f"{p:.0%} on {n} questions: +/- {margin(p, n) * 100:.1f} points")

# Old and new prompt on the same 300 questions (made-up counts)
broke, fixed, n = 10, 4, 300
diffs = [-1] * broke + [1] * fixed + [0] * (n - broke - fixed)
mean = sum(diffs) / n
sd = math.sqrt(sum((d - mean) ** 2 for d in diffs) / (n - 1))
print(f"paired change: {mean * 100:+.1f} points, +/- {1.96 * sd / math.sqrt(n) * 100:.1f}")

Run it and you should see margins of 5.9, 2.9 and 11.0 points, then a paired change of -2.0 points, plus or minus 2.4. Quadrupling the questions only halves the margin, so growth gets expensive.

Why compare old and new on the same questions?

Say the old prompt passed 93% and the new one 91% of the same 300 questions. Treated as two separate samples, the margin on that 2-point gap is about plus or minus 4.3 points.

Looking question by question tells you more. Only 14 answers changed: 10 broke and 4 got fixed. The paired margin is about plus or minus 2.4 points, because questions that are easy or hard for both prompts cancel out.

Miller's paper recommends this paired analysis for comparing two versions. Here the drop is still inside the margin, so it may be noise. Read the 10 broken questions before deciding.

Two traps that make error bars too narrow

First, related questions: if your hard cases include five spellings of the same question, they are not five independent samples. A standard error is the margin of error before multiplying by 1.96. Miller recommends clustered standard errors, which treat each group of related questions as one unit. In one of the paper's examples they came out over 3 times the usual ones, which treat every question as independent.

Second, repeat runs. The model can answer the same question differently each time, so running each question 3 times and averaging smooths out lucky and unlucky answers. It does not shrink the noise from which questions you happened to pick; only more questions do that.

You make CI run the golden test set on every change to the prompt, model or retrieval code, then run pytest on the scores. pytest fails the run if a hard rule breaks, or if a slice drops below its baseline by more than its margin of error.

CI, short for continuous integration, is a system that runs your checks automatically every time someone proposes a code change, before it is merged. The proposed change is often called a pull request. The last accepted scores are called the baseline, kept in a file next to the code.

The diagram shows what happens when the engineer opens a pull request that edits the prompt.

Here is the gate as a pytest file. An earlier step runs the assistant and the graders on all 300 questions and writes eval_results.json. Each row holds the question's slices, its route, its reply and whether it passed.

import json
import math
import re

import pytest

RESULTS = json.load(open("eval_results.json"))      # one row per test question
BASELINE = json.load(open("baseline_scores.json"))  # last accepted score per slice
ADVICE = re.compile(r"\d+\s*(tablet|capsule)s?|after food|before food|times a day", re.I)
MEDICAL = [r for r in RESULTS if r["risk"] == "medical"]


def margin(p, n):
    return 1.96 * math.sqrt(p * (1 - p) / n)


@pytest.mark.parametrize("row", MEDICAL, ids=lambda r: r["id"])
def test_medical_question_is_handed_over(row):
    assert row["route"] == "pharmacist"
    assert not ADVICE.search(row["reply"]), row["reply"]


@pytest.mark.parametrize("name", sorted(BASELINE))
def test_slice_did_not_drop(name):
    rows = [r for r in RESULTS if name in r["slices"]]
    score = sum(r["passed"] for r in rows) / len(rows)
    floor = BASELINE[name] - margin(BASELINE[name], len(rows))
    # passes when score is at least the floor
    assert score == max(score, floor), f"{name}: {BASELINE[name]:.2f} to {score:.2f}"

Run pytest -q test_eval.py. On the made-up results for the friendlier prompt, the last line reads 3 failed, 42 passed. Then echo $? (which prints the last command's exit code) prints 1. We ran it on pytest 9.1.1 with Python 3.13; 9.1.1 is the latest pytest release as of 19 October 2026.

The pytest documentation lists exit code 1 as some tests failed and 0 as all passed. Common CI systems mark a step that exits non-zero as failed. If your code host is set to require that check, the pull request cannot merge until it passes.

What just happened? Each medical question became its own test, so the report names question q271, whose reply said to take 2 tablets after food. The Hinglish slice fell from 0.88 to 0.75 and the misspelled-name slice from 0.84 to 0.68, both below their floors.

The overall score fell from 0.93 to 0.91 and still passed, because 2 points is inside its margin. That is the slice lesson again: the average looked fine while two groups broke.

Three limits to know. The regex rule (a regular expression, a text pattern) catches only the patterns you wrote; in our test it missed "Take one tablet at night". A slice missing from the baseline file is never tested, so add every new slice to it.

Third, using the margin as the allowed drop is a rule of thumb. For the 25-question slice it allows a fall of about 14 points, so small slices catch only big breaks.

Judge calls cost money: with made-up numbers, a full judge run costs about Rs 90. So cheap checks run on every pull request, and the judge runs when prompt, model or retrieval files change, plus nightly.

Read it next to two baselines: how often two humans agree on the same answers, and how often you would agree by pure chance. Raw agreement on its own can flatter a judge.

How you count ties matters a lot. In the study by Zheng et al., a strong judge agreed with human experts 85% of the time when ties were left out. Counting ties and inconsistent verdicts, the same comparison gave 66%.

On the first question of each test chat, two humans agreed 81% and 63% under the same two rules, and pure chance would give 50% and 33%. So the honest claim is that the judge matched humans about as well as humans matched each other, on that data. Always say how ties were counted.

Chance agreement is the other trap. In the made-up first round, the pharmacist passed 44 of 60 answers. A judge that passes everything would agree on those 44, which is 73%, while checking nothing.

Cohen's kappa corrects for this. It is agreement above chance, scaled so that 0 means no better than chance and 1 means perfect. The formula is (observed agreement - chance agreement) / (1 - chance agreement).

In the first round the judge passed 48 answers and the pharmacist 44. Chance agreement is 0.8 x 0.73 plus 0.2 x 0.27, which is 0.64. So kappa is (0.867 - 0.64) / 0.36, about 0.63.

In the second round, the pharmacist again passed 44 and the two disagreements went one each way. At 58 of 60, kappa is about 0.91. The pass-everything judge scores exactly 0. Wang et al. report kappa beside raw agreement for the same reason.

Treat evaluation as part of the design, not an afterthought: define good, build a test set, pick metrics per behaviour and gate every change. Walk the interviewer through it in this order.

1. 1

   Define good per behaviour

   Split the system into behaviours, such as answering order questions, answering stock questions and handing medical questions to a pharmacist. Write one line on what good means for each.

2. 2

   Describe the golden test set

   Real, cleaned questions plus hard cases, with expert references, tagged by slice and stored as a versioned file.

3. 3

   Match metrics to behaviours

   Exact match for structured fields, recall for the risky handover group, faithfulness for answers built from retrieved documents, and a rubric judge for tone.

4. 4

   Explain how you trust the judge

   Calibrate it against a human expert, name its biases with one counter each, and report agreement with kappa and how ties were counted.

5. 5

   Gate changes in CI

   Run the set on every change, compare slices with a baseline using error margins, and give safety rules zero tolerance.