Error Analysis for LLM Apps: Read Traces Before Metrics

The metrics look fine while users complain, because the metrics measure what you guessed would go wrong. LLM error analysis starts from the failures real users actually hit, which no generic score was built to see.

Picture a common setup. A home-services booking app in Hyderabad lets customers book plumbers, electricians and AC technicians through a chat assistant. The setup is made up, but the problem is real.

The assistant handles about 6,000 chats a day. Its dashboard shows a relevance score, where a second model rates each reply from 0 to 1. The average sits at 0.86 and has barely moved in a month.

The operations team sees another story. It took 140 angry calls last week, up from 60 a month earlier. Some customers waited at home for the wrong kind of technician, and others refused a Rs 199 visiting charge nobody had mentioned.

The engineer's first idea is to add more scores: politeness, completeness, maybe a hallucination score (a check for made-up facts). The operations lead suggests something slower and cheaper: sit together for one afternoon and read 120 real chats, start to finish.

That afternoon turns up six kinds of failure, and the relevance score caught almost none of them. A reply that books an electrician for a broken AC is polite and on topic, so it scores well. It is still wrong.

Five words carry the rest of this guide. Each gets a plain meaning and an example from the booking assistant.

• Trace: the full record of one request as it moves through your app. Here one trace holds the customer's messages, the prompt, each tool call and the final reply.
• Span: one step inside a trace, such as one model call or one call to the booking system. OpenTelemetry, an open-source toolkit for recording traces, calls spans the building blocks of traces.
• Tool call: the model asking your code to run an action. The assistant has a tool called find_slots that looks up free technician times; your code runs it and hands back the result.
• Failure category, also called a failure mode: one named kind of mistake that keeps coming back. Example: wrong service type, meaning the assistant booked the wrong kind of technician.
• Eval: an automated check that scores your app's outputs, so you can run it on every change. It can be plain code, or another model acting as a judge.

You do not need any tracing tool to start. A spreadsheet with one chat per row is enough for a first reading session.

LLM error analysis is the habit of reading real outputs from your app and writing down what went wrong in each. You sort those notes into a short list of named failures. Then you count them, fix the biggest, and write checks for them.

Picture a sorter at a big post office in Hyderabad. She does not print bag labels on day one and hope the letters fit. She reads the addresses on a few trays first, and the bags follow from what she sees.

The mapping is direct. Letters are traces, reading an address is writing a note, the bags are failure categories, and the address-unclear pile is your list of notes that fit no category yet.

Where the picture breaks: a letter has one right bag, but one trace can fail in two ways at once. You handle that by noting only the first failure, so each trace still lands in one pile.

Read traces first because you cannot know which metric matters until you have seen how your app actually fails. A metric chosen before reading measures your guesses, and real failures are usually specific to your product.

A 2024 study of people building model-based graders saw this effect directly. The authors call it criteria drift: people need criteria to grade outputs, but grading outputs helps them define the criteria.

The study also reports that some criteria seemed to depend on the specific outputs people had seen. You find those criteria only by reading.

Generic scores also miss your business rules. No ready-made relevance score knows that a gas smell must go to a person within minutes. The operations lead knows it, and she spotted the problem in trace 37.

Here is one trace from the booking assistant. Each box is one span, and each span is a place where something can go wrong.

The final reply looks fine on its own. Only the tool call shows where it went wrong, which is why you read whole traces, not just answers.

Read a mix of random and flagged traces, and keep going until new traces stop showing new kinds of failure. A practitioner FAQ on evals, as of 19 October 2026, gives starting numbers.

That FAQ, by Hamel Husain and Shreya Shankar, says to annotate at least 30 traces yourself before you review suggestions from an AI helper. It calls a working pool of roughly 100 diverse traces a useful guardrail when such a helper joins the review. Treat both numbers as rules of thumb, not proven thresholds. Shankar is also first author of the 2024 study above, so the two are not independent sources.

Random traces show what normal use looks like. Flagged traces find trouble faster. The engineer pulled 120 traces for the first session:

• 60 random chats from the last seven days, so ordinary requests get a fair share.
• 30 chats where the customer tapped thumbs-down.
• 20 chats that ended with a hand-off to the operations team.
• 10 very long chats, over 12 messages, because long chats often mean the assistant is stuck.

Half of this sample was flagged, so it fails more often than everyday traffic does. That is fine for finding and ranking problems. Just do not report the counts as the app's real failure rate.

The stopping point has a name. The same FAQ calls it theoretical saturation: new reviews stop revealing new failure modes or changing the ones you have. The last 30 traces the pair read added no new category, so they stopped at 120.

Write one or two plain sentences saying what went wrong, describing what you saw. This step is called open coding: you label each trace freely, without a fixed list of categories.

The name comes from grounded theory, a method from qualitative research (research that studies words and stories, not numbers). A SAGE encyclopedia entry describes open coding as line-by-line analysis that names as many ideas as possible.

1. 1

   Read the whole trace

   Go from the first message to the last span. Do not judge from the final reply alone.

2. 2

   Mark pass or fail

   Ask one question: would the operations lead be happy if this chat reached a real customer? If yes, mark it pass and move on.

3. 3

   Find the first thing that broke

   Later mistakes often follow from an early one. A wrong service type leads to wrong slots and a wrong price, so note only the first.

4. 4

   Write what happened, not why

   Write that it booked an electrician for an AC that is not cooling. Do not write that the prompt is confusing. Causes come later.

5. 5

   Keep the trace id

   Every note links back to its trace, so anyone can open it and check.

The first-failure rule matches the practitioner FAQ, which says to note the first failure you see because upstream errors can cause downstream ones. The difference between a vague note and a useful one is detail:

• Vague: bad answer. Useful: quoted Rs 349 for a tap repair and left out the Rs 199 visiting charge.
• Vague: made things up. Useful: offered a 7 pm slot, but find_slots returned only 4 pm and 6 pm.
• Vague: did not understand. Useful: the customer wrote in mixed Hindi and English about a dripping tap, and the assistant asked which city they lived in.

Read all your notes, put similar ones together, and give each group a short name and a one-line definition. Aim for a short list that someone else could apply the same way.

Grounded theory calls this step axial coding. A SAGE encyclopedia entry describes it as relating categories to their subcategories. You do not need the name, only the habit of grouping and checking that each note fits one group.

Back at the post office, the sorter watches the address-unclear pile. If it grows large, she opens a new bag for its most common kind. A big pile of other notes means a category is missing.

Grouping by hand can fool you. A 2019 paper on error analysis for NLP models (software that processes human language) warns that subjective sorting of a small sample of errors can give biased and incomplete conclusions. Its three principles, in trace terms:

• Define each category precisely, so two people would put the same note in the same place.
• Look at a large set of examples, including ones that passed, before trusting a pattern.
• Test your explanation of a category before you build a fix on it.

The diagram shows four of the 41 failure notes and where they landed. Follow the arrows from the notes to the tally.

A model can help once the notes exist, for example by suggesting groups for 300 notes or counting them. The practitioner FAQ allows this, but a person who knows the product should still name the categories and check where each note went.

Record one trace per request, with a span for each model call and tool call, and attach the inputs and outputs you will need to judge it. OpenTelemetry gives you this structure.

OpenTelemetry's docs describe it as an open-source framework and toolkit for traces, metrics and logs. The sketch below uses its Python SDK (software development kit, the library you install), version 1.45.1, the latest on PyPI as of 19 October 2026. It keeps spans in memory, an option meant for tests, instead of sending them to a tracing backend (a server that stores and shows traces).

from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter

exporter = InMemorySpanExporter()
provider = TracerProvider()
provider.add_span_processor(SimpleSpanProcessor(exporter))
tracer = provider.get_tracer("booking-assistant")

with tracer.start_as_current_span("chat_turn") as turn:
    turn.set_attribute("booking.prompt_version", "v14")
    with tracer.start_as_current_span("model_call"):
        pass
    with tracer.start_as_current_span("tool.find_slots") as tool:
        tool.set_attribute("booking.service", "electrician")
        tool.set_attribute("booking.slots", "16:00,18:00")

for span in exporter.get_finished_spans():
    trace_id = format(span.context.trace_id, "032x")[:8]
    parent = "root" if span.parent is None else "child"
    print(f"{span.name:<16} trace {trace_id}  {parent}  {dict(span.attributes)}")

Install it with pip install opentelemetry-sdk==1.45.1 and run the file. You should see three lines like these; the 8-character trace id is random on every run:

model_call       trace 32235a5d  child  {}
tool.find_slots  trace 32235a5d  child  {'booking.service': 'electrician', 'booking.slots': '16:00,18:00'}
chat_turn        trace 32235a5d  root  {'booking.prompt_version': 'v14'}

Attributes are key-value details attached to a span. Prefix the names you make up with your app's name, as in booking.service. Avoid prefixes OpenTelemetry already defines, such as the app prefix in app.screen.name; its naming guidance warns they may clash later.

OpenTelemetry also defines shared names for model calls, called the generative AI semantic conventions. As of 19 October 2026, the docs say these conventions have moved to their own repository. Check their stability label there before your dashboards depend on the names.

For error analysis the key attributes are the ones a reader needs: the prompt version, the tool's inputs and outputs, and the reply. Mask personal data before it reaches the store.

Fix first the categories that are common, harmful, or both, especially when the fix is cheap. Count each category, then weigh it by how badly it hurts a customer.

The notes live in a spreadsheet saved as notes.csv, one row per failing trace, with a category column. This plain Python counts them:

import csv
from collections import Counter

with open("notes.csv", newline="", encoding="utf-8") as f:
    notes = list(csv.DictReader(f))   # one row per failing trace

counts = Counter(row["category"] for row in notes)
total_read = 120                      # every trace read, passes and fails

for category, count in counts.most_common():
    print(f"{category:<34} {count:>3}  {count / total_read:.1%}")

With the 41 notes from the first session, it prints this on Python 3.13:

Wrong service type                  12  10.0%
Visiting charge left out             9  7.5%
Mixed Hindi and English misread      7  5.8%
Slot not from find_slots             6  5.0%
Asks a question already answered     4  3.3%
Missed safety hand-off               3  2.5%

Counter tallies how many rows carry each category name, and most_common sorts them from largest to smallest. The share divides by all 120 traces read, not by the 41 failures.

The one-decimal format is deliberate. With no decimals, Python prints 7.5% as 8% but 2.5% as 2%. After multiplying by 100, both land on an exact half, and Python rounds a half to the even digit.

The operations lead put missed safety hand-off first, even though it is the smallest group. Three chats about a gas smell or a sparking switch got a booking for the next day instead of an urgent call.

Wrong service type came second, because it is the most common and wastes a technician's trip. Before fixing it, the engineer tested his explanation, as the 2019 paper advises.

He guessed that AC cases failed because the prompt listed AC repair under electrical work. He moved it to its own line and replayed 20 saved AC chats (made up). Nineteen now booked an AC technician, so the guess held.

Turn each category into a check that runs on every change, using the failing traces you already found as test cases. Use plain code where a rule is exact, and a model judge only where judgement is needed.

• Code checks for exact rules. If a reply quotes a price, it must mention the Rs 199 visit charge. If it offers a slot, find_slots must have returned that slot.
• Saved cases for known failures. The 12 wrong-service chats become test inputs, with the right service type written next to each one.
• A model judge for fuzzy cases. Spotting a safety risk needs reading, so a second model answers yes or no.

Before a check guards releases, test the check itself on replies you already labelled. Here is the visiting-charge check with that test. pytest is a test runner, a tool that finds and runs functions whose names start with test.

import re

def mentions_visit_charge(reply):
    text = reply.lower()
    has_word = "visit" in text
    has_amount = re.search(r"\b199\b", text) is not None
    return has_word and has_amount

# Replies copied from real traces, with the reader's pass or fail label
LABELLED = [
    ("trace-9", "Tap repair is Rs 349. Shall I book 4 pm?", False),
    ("trace-71", "Fan repair costs Rs 299, visit at 6 pm.", False),
    ("trace-12", "Tap repair is Rs 349 plus a Rs 199 visiting charge.", True),
    ("trace-30", "Rs 1990 for a deep clean, including the visit.", False),
]

def test_check_agrees_with_labels():
    for trace_id, reply, expected in LABELLED:
        assert mentions_visit_charge(reply) == expected, trace_id

Save it as test_visit_charge.py and run python -m pytest -q test_visit_charge.py. With pytest 9.1.1 it prints 1 passed in about 0.01 seconds.

The word boundary in the pattern matters. A plain substring test for 199 also matches Rs 1990, and that version fails with AssertionError: trace-30. The trace id in the message points straight back to the chat.

A string check like this is still crude. It also passes a reply saying there is no Rs 199 visiting charge, so keep tricky replies like that in the labelled list as you find them.

In your build pipeline the same function then runs on the replies your current prompt gives to the saved cases. The idea comes from ordinary software testing.

A 2020 paper brought behavioural testing from software engineering to NLP models. Behavioural testing checks what a system does on chosen inputs, without looking inside it.

The authors report that, in their user study, practitioners using its method created twice as many tests and found almost three times as many bugs as those without it.

Compare the judge's answers with a person's labels on the same chats, and count how many real failures it catches, not just how often it agrees. Agreement alone hides rare failures.

The 2024 study above is about exactly this. Its authors argue that model-based graders inherit the problems of the models they grade, so people must check them.

Here the operations lead labels 50 chats for safety hand-offs (made up), and the judge labels the same 50.

Pause and predict before you run it. The judge agrees with her on 47 of the 50 chats. Decide whether you would let it guard safety hand-offs.

human = ["fail"] * 3 + ["pass"] * 47
judge = ["fail", "pass", "pass"] + ["fail"] + ["pass"] * 46

pairs = list(zip(human, judge))
agree = sum(h == j for h, j in pairs) / len(pairs)
caught = sum(h == j == "fail" for h, j in pairs) / human.count("fail")
cleared = sum(h == j == "pass" for h, j in pairs) / human.count("pass")
print(f"agreement {agree:.0%}, failures caught {caught:.0%}, passes cleared {cleared:.0%}")

It prints this line:

agreement 94%, failures caught 33%, passes cleared 98%

If you treat a failure as the positive case, failures caught is the true positive rate (also called recall), and passes cleared is the true negative rate. Track both.

With only 3 real failures, each one moves the caught rate by 33 points. Label more failing chats before you trust the number, then improve the judge's prompt until it catches them.

The person who knows what a good answer looks like should read, with an engineer beside them. Read again after every real change and on a fixed schedule, because new failures keep appearing.

The practitioner FAQ says a domain expert should do this step. Here the operations lead reads because she knows the business rules: a geyser that trips the switch is an electrician's job, and a gas smell is urgent.

The engineer reads with her because he knows which span to open when a reply looks odd. Keep one person in charge of the category names, because two naming styles make the counts meaningless.

Once a month, both read the same 20 traces separately and compare. Where they disagree, they sharpen the definition.

Do not hand all the reading to a model. It can draft notes, but it does not know the company never sends technicians after 8 pm. If nobody reads, nobody learns what the criteria should be.

Here is the loop the team now runs every Monday. Start at the top and follow it round.

The loop keeps finding things. When the monsoon arrived, a new category showed up: waterproofing requests booked as plumbing. No existing metric flagged it, but the second Monday reading did.

Categories can retire too. When a check has passed for four weeks in a row, the team keeps the test but stops counting that category by hand. Reading 50 traces takes the pair about 90 minutes a week (made up).

Explain it as the step that decides what you measure: log traces, read them, name the failures, count them, then build evals for the top ones.

1. 1

   Say what you log

   Every request becomes a trace: messages, prompt version, model and tool calls with inputs and outputs, reply, cost and time. Personal data is masked before storage.

2. 2

   Describe the sample

   Random traces show normal use, and flagged ones such as thumbs-down chats and hand-offs find failures fast. Read until new traces stop adding categories.

3. 3

   Name failures from the data

   Write a note on the first failure in each trace, then group the notes into a short list of categories with one-line definitions.

4. 4

   Prioritise

   Rank categories by how often they happen and how much harm they do. Safety failures go first even when rare.

5. 5

   Close the loop

   Each category becomes a code check, a set of saved cases or a judge checked against human labels on failures caught. Repeat after every release.