Multi-Turn Evaluation: Testing Whole LLM Conversations

It can still fail real users, because people spread their needs across many messages. Single-turn tests check one message at a time, so they never look at the gaps between messages.

Multi turn evaluation is how you test those gaps. This guide shows what to test, how to fake a user, how to score a whole chat and how many times to run each test.

Picture a common setup. A property website runs a chat assistant that helps people in Pune shortlist rental flats. You tell it your budget, area, flat size and house rules, and it suggests listings. The setup is made up, but the problem is real.

The team's test suite has 500 cases. Each case is one message, such as a request for a 2BHK flat in Baner under Rs 25,000, plus a check on the reply. A 2BHK has two bedrooms, a hall and a kitchen.

The suite passes 96% of cases, and everyone feels safe shipping. Then the conversation designer reads 300 saved chats that ran longer than five messages. About one in seven goes wrong somewhere in the middle. Here is a typical one, turn by turn.

• Turn 1: A renter asks for a 2BHK in Baner under Rs 25,000. The assistant shows three flats, all within budget.
• Turn 2: She adds that she has a dog. The assistant narrows the list to the two pet-friendly flats.
• Turn 3: She asks whether Wakad has anything similar. The assistant shows three Wakad flats: one costs Rs 31,000 and one bans pets.
• Turn 4: She asks which flats she can visit on Saturday. The assistant suggests the Rs 31,000 flat and the one that bans pets.

Read turn 3 on its own and it looks like a sensible answer about Wakad. The mistake is that the assistant dropped the budget and the dog when the area changed. None of the 500 tests could notice, because none of them had a turn 2.

A single-turn test of turn 3 sees Wakad flats for a Wakad question and passes. Only a check that carries the rules from turns 1 and 2 forward can catch the miss.

You need seven words, each shown on the renter's chat. An LLM, or large language model, is the AI that reads and writes the text inside the assistant. A turn is one user message plus the assistant's reply.

A conversation is the full chain of turns. The context is everything earlier in it that the assistant should still respect, such as the Rs 25,000 budget. A transcript is the saved record of a conversation, turn by turn.

A scenario is the written plan for one test conversation: who the user is, what they want and when they reveal each detail. A benchmark is a fixed public set of tests that many teams run, so their results can be compared.

Multi turn evaluation is testing a chat assistant on complete conversations, where each reply must respect everything said before it. You still check single replies, but always against the full history, and you grade the conversation as a whole.

Think of chess puzzles. A puzzle gives you one board position and asks for the single best move. Solving a thousand puzzles shows you can spot tactics, not that you can play a whole game.

Single-turn tests are chess puzzles: one board position, the user's message, and one move to check, the reply. Multi-turn evaluation makes the assistant play the whole game, from the opening message to a shortlist the user can act on.

Where the picture breaks: in chess the rules never change mid-game. A real user can change the rules, such as raising the budget in turn 5. A good multi-turn test checks that the assistant follows that change.

Researchers have built public benchmarks around this idea. MT-Bench-101, published at ACL 2024, a major language research conference, holds 1,388 multi-turn dialogues with 4,208 turns across 13 tasks.

Its authors tested 21 models. They found that common alignment techniques, the ways models are tuned to follow instructions and chat, did not clearly improve multi-turn ability.

Single-turn tests miss chat failures because each test starts from an empty conversation. Nothing earlier exists, so nothing can be forgotten, contradicted or corrected. The failure needs history, and the test has none.

Look at what the 500 cases have in common. Every case states the whole request in one clean message. Real users rarely do that: this renter revealed her budget, her dog and her second area across three messages.

A 2025 study, LLMs Get Lost In Multi-Turn Conversation, measured this directly. The authors took tasks that work as one complete instruction and split each into shards, small pieces of information revealed one per turn. Together the shards held the same information as the original.

Pause and predict: if a model gets exactly the same information, only spread over several turns, how much should its score change? Ideally, not at all.

The authors ran 15 models through more than 200,000 simulated conversations. The six task types were code, database queries, API calls (requests one program sends to another), maths, turning structured data into text, and summaries. On average, scores fell 39% when the instruction arrived in shards.

The paper splits that drop into two parts. Aptitude, the best-case score (the 90th percentile, the score that 9 in 10 runs fall at or below), fell about 16%, which the authors call minor. Unreliability, the gap between the 90th and 10th percentile runs, rose about 112%, more than doubling.

In plain words, the models could mostly still solve the task. What changed is that the same task now succeeded on some runs and failed on others.

The authors trace much of the drop to early guessing. Models assume missing details, attempt a full answer too soon and then lean on that attempt. In the paper's words, models that take a wrong turn "get lost and do not recover."

That matches the renter's chat. In turn 1 the assistant built its search around Baner. When she asked about Wakad, it treated the new area as a fresh search and forgot the rest.

The failures that need several turns are about memory, consistency and recovery. They cannot happen in a one-message test, so a team that only runs single-turn tests has never measured them.

Sorting the 300 long chats, the team found six recurring kinds of failure. Your assistant will have its own list, but these six are common in chat products.

• Forgotten constraint. A rule from an early turn, like the Rs 25,000 budget, is dropped later.
• Ignored correction. The user says the office is in Hinjewadi, not Kharadi, and the assistant keeps suggesting Kharadi flats.
• Contradiction. In turn 2 a flat has covered parking, and in turn 6 the same flat has none.
• Early guessing. The user only says they want a flat, and the assistant assumes a 1BHK and builds on that guess.
• Repeating itself. The assistant shows the same three flats after the user rejected them.
• Losing the goal. After a long side chat about deposits, it never returns to the shortlist the user wanted.

Most of these can be checked mechanically. A forgotten budget is a rent above a limit. A repeated flat is an id that comes back after a rejection. Only a few, like losing the goal, need judgement.

You write a scenario: a short plan describing the user, their goal, the details they reveal in each turn, and the rules a correct conversation must follow. The scenario is the test case, and each conversation is one run of it.

The conversation designer writes the scenarios, because she knows how real users talk. An evaluation engineer turns them into code. The scenario for the dog owner's chat has five parts.

1. 1

   Describe the user

   A renter moving to Pune for work, polite but brief. She types short messages and reveals one need at a time.

2. 2

   State the goal

   A shortlist of at least two 2BHK flats she could visit this weekend.

3. 3

   List what she reveals and when

   Baner and Rs 25,000 in turn 1, the dog in turn 2, Wakad in turn 3. Each reveal adds a rule the assistant must keep.

4. 4

   Write the success rules

   Every suggested flat fits the latest budget, pet rule and areas. The final message holds a shortlist. No rejected flat comes back.

5. 5

   Add a twist

   In turn 5 the renter raises her budget to Rs 28,000. The test checks that the new limit replaces the old one.

The twist matters. A test where the user only adds rules checks memory. A test where the user changes a rule also checks that the assistant updates its memory instead of clinging to the first version.

Checking the rules in code

The team stores each run as a list of turns. Each turn holds new_rules, the rules the scenario reveals in that turn, plus suggested_flats, the flats the assistant actually showed. The function walks the conversation in order.

The sample below includes the renter's chat as test data. Save it as check_rules.py and run python3 check_rules.py. We ran it on Python 3.13; it needs no extra packages. You should see exactly the four lines shown after the code.

def check_rules(turns):
    problems = []
    rules = {}
    for i, turn in enumerate(turns, start=1):
        rules.update(turn.get("new_rules", {}))  # what the user has asked for so far
        for flat in turn.get("suggested_flats", []):
            if flat["rent"] > rules.get("max_rent", float("inf")):
                problems.append(f"Turn {i}: {flat['id']} is over budget")
            if rules.get("pets") and not flat["pets_ok"]:
                problems.append(f"Turn {i}: {flat['id']} does not allow pets")
            if "areas" in rules and flat["area"] not in rules["areas"]:
                problems.append(f"Turn {i}: {flat['id']} is outside the asked areas")
    return problems


# The renter's chat: rules come from the scenario, flats from the assistant's replies
chat = [
    {"new_rules": {"max_rent": 25000, "areas": ["Baner"]},
     "suggested_flats": [{"id": "B1", "rent": 24000, "pets_ok": True, "area": "Baner"},
                         {"id": "B2", "rent": 23500, "pets_ok": False, "area": "Baner"},
                         {"id": "B3", "rent": 25000, "pets_ok": True, "area": "Baner"}]},
    {"new_rules": {"pets": True},
     "suggested_flats": [{"id": "B1", "rent": 24000, "pets_ok": True, "area": "Baner"},
                         {"id": "B3", "rent": 25000, "pets_ok": True, "area": "Baner"}]},
    {"new_rules": {"areas": ["Baner", "Wakad"]},
     "suggested_flats": [{"id": "W1", "rent": 22000, "pets_ok": True, "area": "Wakad"},
                         {"id": "W2", "rent": 31000, "pets_ok": True, "area": "Wakad"},
                         {"id": "W3", "rent": 21000, "pets_ok": False, "area": "Wakad"}]},
    {"suggested_flats": [{"id": "W2", "rent": 31000, "pets_ok": True, "area": "Wakad"},
                         {"id": "W3", "rent": 21000, "pets_ok": False, "area": "Wakad"}]},
    {"new_rules": {"max_rent": 28000},
     "suggested_flats": [{"id": "W1", "rent": 22000, "pets_ok": True, "area": "Wakad"},
                         {"id": "B4", "rent": 27500, "pets_ok": True, "area": "Baner"}]},
]
for problem in check_rules(chat):
    print(problem)

Turn 3: W2 is over budget
Turn 3: W3 does not allow pets
Turn 4: W2 is over budget
Turn 4: W3 does not allow pets

What just happened? The rules.update line merges each turn's new rules into everything asked so far. So the Wakad turn adds an area but keeps the budget and the dog, and every flat is checked against all of them.

The checks flag turn 3 first, before the renter noticed anything in turn 4. Turn 5 passes: the Rs 27,500 flat fits because the raised budget replaced the old one. Note that the scenario lists both areas in turn 3, since update replaces a key rather than adding to it.

The rules come from the scenario, never from the assistant's own summary. If the assistant misremembers the budget, its summary is wrong too, and the test would agree with the mistake.

You use a simulated user: a second model that plays the person in your scenario and writes each user message in reply to the assistant. One scenario then produces many natural, slightly different conversations.

Scripted users are the simple option. You write every user message in advance and send them in order. Scripts are cheap and repeatable, but they break when the assistant asks a question the script did not expect.

Suppose the assistant asks the renter whether she wants a furnished flat. A script has no answer and sends its next line about Wakad. The conversation stops making sense, and the test result means little.

A simulated user reads the scenario card and the chat so far, then writes its next message in character. It can answer the furnished question, because the card says she does not mind either way.

The 2025 sharding study worked this way. Its simulated user saw every shard, revealed at most one per turn and reworded each in its own words. Turn 1 always carried the main request.

Tau-bench, a 2024 benchmark for assistants that use tools, applies the same idea to customer service. A model plays customers of a retail shop or an airline, each following a written task instruction. The chat ends when the simulated customer sends a stop signal.

Tau-bench then scores each chat with two checks. The database at the end must match the expected final state, and the assistant's replies must contain any information the task requires.

Here is the loop the rental team built. Start at the scenario card. The question box decides whether the conversation goes round again or gets scored.

A simulated user brings its own risks. The tau-bench authors list unclear task instructions, weak reasoning and a simulated user that does not always follow its instructions. Beyond that list, a simulated user can also reveal everything in turn 1 or quietly help the assistant.

So have a person read a sample of simulated chats every week. The question is simple: do the fake users still behave like the real ones in your saved chats?

You score it at two levels: checks after every turn, and a verdict for the whole conversation. Hard rules are checked in code, and softer qualities are scored by a judge following a written rubric.

Turn-level checks catch the exact moment something broke, like turn 3 in the renter's chat. The conversation-level verdict answers the question users care about: did I end up with a shortlist I could use?

An LLM judge is a model asked to grade a transcript against your rubric, a short list of what good looks like. The 2023 paper that introduced MT-Bench, a set of 80 questions with two turns each, studied such judges.

The graders were expert humans, mostly graduate students. When picking the better of two answers, with ties left out, the strongest judge agreed with them 85% of the time, and two humans agreed 81% of the time. With ties counted, both fell, to 66% and 63% on first turns.

The same paper studied judge biases. Position bias means favouring one position, such as the answer shown first. Verbosity bias means favouring longer answers. Self-enhancement bias means favouring answers the judge itself wrote, but the authors said their data could not settle it.

Two of its recommendations matter for multi-turn work. First, ask the judge twice with the two answers swapped, and treat a disagreement as a tie. Second, show the judge the whole conversation in one prompt; split into separate prompts, it struggled to find the earlier replies.

So the team splits the work. Code decides budget, pets, area and repeats, where it is exact and costs nothing. The judge scores only three things: were questions clear, did the tone stay friendly, and was the final shortlist explained?

The table compares the two kinds of test on the rental assistant. The costs and run times are made up but realistic for a small team.

The two kinds of test work together. Keep the fast single-turn suite for every small change, and run the conversation suite before each release.

Run each scenario several times, often 5 to 10, because the same scenario can pass on one run and fail on the next. One pass shows the assistant can succeed, not that it usually will.

Chat tests vary for two reasons. The model picks its words with some randomness, and a simulated user words things differently each run. Temperature is the setting that controls that randomness.

Turning it down does not fix the problem. The sharding study tested this on two models, with temperature 0 for both assistant and simulated user. In sharded chats, the unreliability gap between the 90th and 10th percentile runs stayed around 30 points out of 100. At temperature 0, the gap for single-message instructions fell below 10.

Tau-bench measures this with pass^k, read as pass hat k. It is the chance that k fresh runs of the same task all succeed. The familiar pass@k is the chance that at least one of k runs succeeds, which rewards a lucky run.

The gap can be large. In the June 2024 tau-bench paper, the best agent tested passed over 60% of retail tasks on average, yet its retail pass^8 fell below 25%. Airline tasks were harder, at about 35% on average.

The team runs every scenario 10 times and reports the pass rate, pass@3 and pass^3. Here is the calculation, using made-up results for three scenarios. It runs on Python 3.8 or later and prints the output shown after it.

from math import comb

def pass_hat_k(passes, runs, k):
    """Chance that k fresh runs of one scenario ALL pass."""
    return comb(passes, k) / comb(runs, k)

def pass_at_k(passes, runs, k):
    """Chance that AT LEAST ONE of k fresh runs passes."""
    return 1 - comb(runs - passes, k) / comb(runs, k)

results = {"pets_then_budget": 9, "change_area": 6, "correct_the_bot": 10}
for name, passes in results.items():
    print(f"{name:17} rate {passes / 10:4.0%}  pass@3 {pass_at_k(passes, 10, 3):.2f}  pass^3 {pass_hat_k(passes, 10, 3):.2f}")

pets_then_budget  rate  90%  pass@3 1.00  pass^3 0.70
change_area       rate  60%  pass@3 0.97  pass^3 0.17
correct_the_bot   rate 100%  pass@3 1.00  pass^3 1.00

The comb function counts the ways to pick 3 runs. pass^3 divides the ways to pick 3 from the passing runs by the ways to pick 3 from all 10. When fewer than 3 runs passed, comb returns 0, so no special case is needed.

What just happened? The change_area scenario passed 6 runs in 10. Its pass@3 of 0.97 says it nearly always succeeds once in three tries. Its pass^3 of 0.17 says three users in a row all get a correct chat about one time in six.

Less sure than the numbers suggest. Ten runs give a rough estimate, and a perfect score can hide a real failure rate.

Pause and predict: a scenario truly passes 90% of the time. In a batch of 10 runs, how often does it score a perfect 10 out of 10?

About 35% of the time, because 0.9 to the power 10 is 0.349. Our simulation of 100,000 batches gave 0.347. So in roughly one release check in three, this flaky scenario looks perfect.

It works the other way too. A score of 9 out of 10 fits a true pass chance anywhere from about 60% to 98%. That is a 95% Wilson interval, a standard way to put a range around a pass rate measured from few runs.

More runs narrow the range. With 90 passes in 100 runs, the 95% range shrinks to 83% to 94%. Spend extra runs on the scenarios that guard your most expensive failures.

Three details that change the number

First, use the comb formula, not the pass rate to the power k. For a true 90% pass chance, the true pass^3 is 0.729. Averaged over many 10-run batches, the comb formula gives exactly 0.729, while the pass rate cubed gives 0.753.

Second, the formula assumes each run is independent, like separate coin tosses. Tau-bench defines pass^k over independent, identically distributed trials. A shared response cache or a fixed seed for the simulated user can make runs repeat each other and break that assumption.

Third, you need at least k runs. With fewer runs than k, comb(runs, k) is 0 and the function divides by zero. Plan at least k runs per scenario, and preferably many more.

You replay the transcript turn by turn and find the first turn where a check failed or the assistant's understanding went wrong. That turn, not the last one, is where the fix belongs.

Chess players do this after a lost game. They replay the moves and mark the first real mistake. A checkmate at move 40 often started with a weak move at move 12.

In the renter's chat, she noticed the problem at turn 4, but the checks flag turn 3 first. That is where the assistant rebuilt its filters for Wakad from scratch. Fixing only the turn 4 reply would hide the symptom.

The team's report lists the first failing turn for every failed run, grouped by failure type. The conversation designer then reads those turns together, which shows patterns no single transcript reveals.

• Save the full transcript of every run, including the assistant's tool calls, such as each listings search it ran.
• Record which scenario rule each check tested, so a failure points back to the scenario.
• Tag the first failing turn, not just the final verdict.
• Turn every real bug into a new fixed scenario, so it stays fixed.

After the first round of fixes, the team changed how the assistant stores rules between turns. The over-budget failure in the area-change scenario fell from 4 runs in 10 to 0 in 10 (made up).

Zero failures in 10 runs still fits a true failure rate of up to about 28%, the 10 out of 10 row in the table above. So the scenario stays in the suite and keeps running before every release.

Explain it as testing whole conversations instead of single replies. Then walk through the scenario, the simulated user, the checks and the repeated runs, and tie each part to a failure it catches.

1. 1

   Start with the gap

   Say that single-turn tests start from an empty chat, so they cannot catch forgotten rules, ignored corrections or contradictions.

2. 2

   Define a scenario

   A user, a goal, details revealed turn by turn, and success rules. Mention one twist where the user changes their mind.

3. 3

   Choose the user side

   Scripts to recheck old bugs exactly, a simulated user for variety. Say that someone must check the simulated user stays in character.

4. 4

   Score at two levels

   Code checks hard rules after every turn. A judge with a rubric scores softer qualities, reading the whole chat in one prompt.

5. 5

   Measure reliability

   Run each scenario several times and report pass^k next to the pass rate, with the number of runs behind both.

6. 6

   Close the loop

   Log the first failing turn, fix the cause, and add every real bug as a new scenario.

Here is the full test setup the team runs before each release. Each box is one part of the system, and its note says what it does.

The frozen listings matter. If the real listings change between runs, a failure might come from a new flat, not from the assistant. A fixed copy keeps every run comparable.