LLM Observability: Catching Silent Regressions and Cost Drift

Nothing looks broken: every request succeeds and no alarm fires, but the answers get worse and often the bill rises too. This is a silent regression: normal error monitoring cannot see it, but LLM observability can.

Picture a common setup. A telecom company's app has an AI chat assistant that answers questions about recharge plans, bills and data packs. It handles about 40,000 conversations a day. The setup is made up, but the problem is real.

One Monday, an engineer changes the assistant's instructions. He pastes the full list of recharge plans into them, so the assistant will "know every plan". He also adds one line: "Explain your answer step by step." He tries ten questions, likes the answers and ships it as prompt version 15 (v15).

Every request returns success. No error alert fires, and the servers look healthy.

Three weeks later, two teams notice trouble. Support sees 17 in every 100 customers ask for a human, up from 11. Customers are quoting plan prices that changed months ago. Then finance asks why the daily AI bill rose 40 percent, from Rs 16,000 to Rs 22,400.

Two things went wrong at once. The answers got worse, which is a silent regression. The cost per conversation crept up, which is cost drift.

LLM observability is the set of records and checks that shows a rising bill within hours and worse answers within a day, then points to the cause. This guide builds it for the team, one piece at a time.

You need six words to follow this guide. Each gets a plain meaning and an example from the telecom team.

Model and prompt. The model is the AI itself: a program that reads text and writes text back. The prompt is everything your program sends it, including fixed instructions and the customer's message. The team's v15 prompt holds the instructions, the plan list and the question.

Token. A token is a small piece of text, often a word or part of a word. Hosted model services usually charge per token, with one price for tokens going in and often a higher one for tokens coming out. The pasted plan list added about 700 input tokens to every call.

Log record. A log record is one saved set of facts about one event. The team writes one record per model call, with its tokens, time taken and cost in rupees.

Version. A version is a label for one exact setup: the prompt text, the model name and its settings. If every record says prompt_version v15, the team can compare days on v14 with days on v15.

Drift. Drift is a slow change in quality or cost over time. Quality drift is the silent regression; cost drift is the bill creeping up. Both of the team's problems started the day v15 shipped.

Observability. The OpenTelemetry project defines it as understanding a system from the outside by asking it questions, including ones you did not plan for. A good test question: did answers about plans get worse after v15 while answers about bills stayed fine?

LLM observability starts with one record per model call that holds its tokens, time, cost, result, tool calls and the exact versions in use. Leave any of these out, and some question about drift becomes impossible to answer later.

Here is what the team records for every call:

• Who and where: a conversation ID, a call ID and a scrambled customer ID, never the phone number. Also the feature that made the call, such as plan questions or bill questions.
• Versions: the prompt version, the model name you asked for, and the model name the service reports back. Also settings such as temperature (how random the wording is) and the output limit.
• Size: input and output tokens, plus cached tokens (repeated prompt text that some services charge less for) if your service reports them.
• Time (latency): total time taken, and time to first token, which is how long the customer waits before the first word appears.
• Cost: rupees for this one call, worked out from the tokens and your price list.
• Result: the finish reason (why the model stopped: it was done, it hit the length limit, or it asked for a tool) and any error.
• Tool calls: each tool's name, time taken and whether it worked. A tool is an action your code runs when the model asks, such as looking up a bill.
• Content: not the raw text, but a reference to the redacted prompt and reply in a separate, locked store. The privacy section below explains why.

The team puts a small wrapper around every model call, so nobody can forget a field. The file below runs on plain Python 3 with nothing to install. A stand-in replaces the real model service, and the prices are made up.

import json, time
from types import SimpleNamespace

PROMPT_VERSION = "v15"
MODEL_NAME = "chat-model-large"
PRICE_IN, PRICE_OUT = 75, 250  # made-up rupees per million tokens

def call_model(messages, model):
    # Stand-in for your provider's client, so this file runs offline.
    return SimpleNamespace(model=model + "-2026-09", input_tokens=2700,
                           output_tokens=310, finish_reason="stop")

def call_model_logged(messages, conversation_id, feature):
    start = time.monotonic()
    reply = call_model(messages, model=MODEL_NAME)
    cost = (reply.input_tokens * PRICE_IN + reply.output_tokens * PRICE_OUT) / 1_000_000
    print(json.dumps({
        "conversation_id": conversation_id,
        "feature": feature,
        "prompt_version": PROMPT_VERSION,
        "model_requested": MODEL_NAME,
        "model_returned": reply.model,
        "input_tokens": reply.input_tokens,
        "output_tokens": reply.output_tokens,
        "latency_ms": round((time.monotonic() - start) * 1000),
        "cost_rs": round(cost, 4),
        "finish_reason": reply.finish_reason,
    }, indent=1))
    return reply

call_model_logged([{"role": "user", "content": "Why is my bill Rs 649?"}], "c-101", "bills")

Run it with python3 and you should see this record, with latency_ms near 0 because the stand-in answers instantly:

{
 "conversation_id": "c-101",
 "feature": "bills",
 "prompt_version": "v15",
 "model_requested": "chat-model-large",
 "model_returned": "chat-model-large-2026-09",
 "input_tokens": 2700,
 "output_tokens": 310,
 "latency_ms": 0,
 "cost_rs": 0.28,
 "finish_reason": "stop"
}

What just happened? The wrapper timed the call, priced it and printed one record before handing the reply back unchanged. Check the cost by hand: 2,700 times 75 plus 310 times 250, divided by one million, is 0.28 rupees. Real code should also write a record when the call fails, with the error's name.

Notice the two model names. The name you send can be a short alias, and the service may report the exact model behind it. The OpenTelemetry conventions keep separate attributes for both names. In a real app, the print line becomes a write to your log store.

A trace is the record of one request as it moves through your system, and a span is one timed step inside it. Together they show which step of a multi-step answer was slow, costly or wrong.

One customer question is rarely one model call. Say a customer asks why her bill is Rs 649 when her plan is Rs 499. The assistant loads her chat history and asks the model what to do.

The model asks for a tool called get_bill. A second model call writes the answer, and a safety check reads it before she sees it.

Think of tracking a courier parcel. The tracking page is the trace: the whole journey of one parcel. Each line on it, such as picked up or reached the Howrah hub, is a span with its own time. The tracking number shared by every line is the trace ID.

Where the picture breaks: parcel stops happen one after another, but spans can overlap, run side by side, and hold smaller spans inside them. A tracking line also has no price, while a span can carry its token count and cost.

Every span has a start time, an end time and attributes, which are small key and value facts such as input_tokens 2700. The top span, called the root span, covers the whole question. The steps inside it are child spans, and all of them share one trace ID.

This one trace already shows something the old logs could not. Both model calls carry the 700-token plan list, though the customer asked about a bill. A question that used to need two short calls now needs two long ones.

Pass the trace context along when you call other services; OpenTelemetry calls this context propagation. If the billing service behind get_bill records spans with the same trace ID, they join the same trace. One search then shows everything that happened for that question.

Create one root span per question and one child span per model call or tool run, and name their attributes with the OpenTelemetry generative AI conventions. Tracing tools that understand these shared names can then chart your tokens and models without custom mapping.

OpenTelemetry is an open-source project, not tied to one company, that produces traces, metrics (numbers measured over time) and logs. Its semantic conventions are agreed names for common facts, so every team writes input tokens the same way.

For a model call span, the client inference page of the conventions defines these names, among others:

• gen_ai.operation.name: the kind of work, such as chat for a model call or execute_tool for a tool run. It is required.
• gen_ai.provider.name: who serves the model. It is also required; the example below leaves it out to stay vendor-neutral, so add it in real code.
• gen_ai.request.model and gen_ai.response.model: the model name you asked for and the one that actually answered.
• gen_ai.usage.input_tokens and gen_ai.usage.output_tokens: the token counts. The input count should include cached tokens.
• gen_ai.prompt.name and gen_ai.prompt.version: the name and version of your prompt, when you use a named prompt template.
• gen_ai.response.finish_reasons: why the model stopped, as a list, such as stop or length.
• gen_ai.conversation.id: the conversation the call belongs to, when your app has such an ID.

A model call span should be named after the operation and the requested model, such as chat followed by the model name. A tool span should be named execute_tool followed by the tool name, such as execute_tool get_bill. A model call to another service should also have the span kind CLIENT, which marks a call out of your program.

The file below records the first model call and the tool run of the bill question. It needs the opentelemetry-sdk package; we ran it on version 1.45.1. The in-memory exporter keeps finished spans in a list, so you can print them instead of sending them to a server.

from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter
from opentelemetry.trace import SpanKind

exporter = InMemorySpanExporter()
provider = TracerProvider()
provider.add_span_processor(SimpleSpanProcessor(exporter))
tracer = provider.get_tracer("chat-service")

with tracer.start_as_current_span("chat_turn"):
    with tracer.start_as_current_span("chat chat-model-large", kind=SpanKind.CLIENT) as span:
        span.set_attribute("gen_ai.operation.name", "chat")
        span.set_attribute("gen_ai.request.model", "chat-model-large")
        span.set_attribute("gen_ai.prompt.name", "telecom-assistant")
        span.set_attribute("gen_ai.prompt.version", "v15")
        span.set_attribute("gen_ai.usage.input_tokens", 2700)
        span.set_attribute("gen_ai.usage.output_tokens", 40)
    with tracer.start_as_current_span("execute_tool get_bill") as span:
        span.set_attribute("gen_ai.operation.name", "execute_tool")
        span.set_attribute("gen_ai.tool.name", "get_bill")

for s in exporter.get_finished_spans():
    trace_id = format(s.context.trace_id, "032x")
    role = "child" if s.parent else "root"
    print(s.name, trace_id[:8], role)

Run it and you should see three lines like these. Your 8-character trace ID will differ, but it must be the same on all three lines:

chat chat-model-large 9be6109e child
execute_tool get_bill 9be6109e child
chat_turn 9be6109e root

What just happened? Each with block opened a span and closed it at the end of the block. Spans are listed in the order they finished, so the root comes last. The shared trace ID is what lets a tracing tool rebuild the tree.

Prompt and reply text is off by default

The conventions say instrumentations, the code or libraries that create spans for you, should not capture prompts, instructions or replies by default. This text is often large and can be sensitive. Capturing it is opt-in, which means you must switch it on.

The conventions list three patterns. The first, and the default, records no text. The second records text on the spans, which suits cases like testing before release, where volume is manageable and privacy rules are met.

For production, where telemetry volume (the amount of traces, metrics and logs sent out) or sensitive data is a concern, they recommend the third. Store the text elsewhere and record only a reference on the span, so that store can have its own, tighter access rules. As of 22 October 2026, the conventions name no standard attribute for that reference yet, so choose your own.

Remove personal data before anything is written, and keep conversation text in a separate store with tight access and a short life. Put only numbers, versions and references in the logs the whole team can read.

Customers type phone numbers, Aadhaar numbers (India's 12-digit identity number), email addresses and home addresses into the chat. If those went straight into logs, every engineer with log access could read them.

The OWASP Logging Cheat Sheet, from a non-profit that publishes web security guidance, lists data that should usually not be logged directly. The list includes session IDs (codes that keep a user signed in), access tokens, passwords, payment card data and sensitive personal data. It says such data should be removed, masked, sanitised, hashed or encrypted. A hash turns text into a fixed-length code, and the same text always gives the same code.

Redaction means replacing personal data with a placeholder before saving. Here is the team's first redaction step. It uses regular expressions, which are text patterns that Python's re module can search for.

import re

PATTERNS = {
    "[PHONE]": re.compile(r"(?:\+91[\s-]?|\b0|\b)[6-9]\d{4}[\s-]?\d{5}\b"),
    "[AADHAAR]": re.compile(r"\b\d{4}[\s-]?\d{4}[\s-]?\d{4}\b"),
    "[EMAIL]": re.compile(r"\b[\w.+-]+@[\w-]+(?:\.[\w-]+)+\b"),
}

def redact(text):
    for label, pattern in PATTERNS.items():
        text = pattern.sub(label, text)
    return text

print(redact("Call me on 9830012345 or mail [email protected]"))
print(redact("+91 98300 12345 and 09830012345"))
print(redact("Aadhaar 2345 6789 0123"))

Run it with python3. You should see three lines: "Call me on [PHONE] or mail [EMAIL]", then "[PHONE] and [PHONE]", then "Aadhaar [AADHAAR]".

The phone pattern finds a 10-digit Indian mobile number starting with 6, 7, 8 or 9. It also catches a leading 0 or +91, and one space or dash after the fifth digit. The Aadhaar pattern finds 12 digits, with or without a space or dash after every four. It also hides other 12-digit numbers, which is the safe kind of mistake.

The team also follows these rules, most of them from the same OWASP page:

• Replace the customer ID with a keyed hash, such as HMAC-SHA256 with a secret key kept out of the logs. Calls can then be grouped by customer without showing who it is.
• Avoid a plain hash of a phone number. There are only 4 billion possible 10-digit numbers starting with 6 to 9, so anyone can hash them all and match yours.
• Keep two stores. Tokens, cost and versions go where the whole team can read them. Redacted text goes to a locked store, kept for 30 days (made up), where every read is recorded.
• Never log secrets: API keys (the secret codes your program uses to call a service), session IDs, passwords or one-time passwords.
• Strip line breaks (carriage returns and line feeds, the hidden characters that start a new line) from customer text. Otherwise someone could type a fake log line into the chat.
• Follow the retention period (how long you may keep data) that your policy and the law set, and delete old text on time.

Watch cost and tokens per conversation, not only the monthly bill, and alert when they move away from their recent normal. Put latency, errors and quality signals on the same dashboard, and split every chart by prompt and model version.

The total bill can rise for a good reason, such as more customers. Cost per conversation rising while traffic stays flat is drift. That is why ratios matter more than totals.

Here is the team's cost maths, with a made-up price list of Rs 75 per million input tokens and Rs 250 per million output tokens. Each conversation makes about two model calls. One rupee is 100 paise.

Input tokens grew 35 percent and output tokens 55 percent, so at these prices cost per conversation rose 40 percent. The v15 row matches the cost_rs value the wrapper printed earlier.

One more term before the next table. The p95 latency is the time within which 95 out of every 100 requests finish. It shows what your slowest customers feel, which an average hides.

With these rules, v15 would have fired the cost alert about 2 hours after going live. Cost per conversation jumped from 40 to 56 paise, far above the 46-paise limit, while traffic stayed flat. The team would have known on day one, not in week three.

They score a small random set of real, live conversations every day, which turns answer quality into a number you can chart and alert on. Scoring live traffic like this is called online evaluation.

Offline evaluation is different: you run a fixed set of test questions before a release. The engineer's ten test questions were a tiny offline check, and they all looked fine. Live traffic holds kinds of questions that no small test set covers.

The team samples 2% of conversations, about 800 a day. Sampled chats are scored three ways:

• A code check: every price and plan name in the reply must exist in today's plan list. Code checks are cheap and give the same answer every time.
• A judge model: a second model reads the conversation with a short checklist and returns pass or fail with a one-line reason.
• People: the support team reads 50 sampled chats a day, including every fail and some random passes, to check that the judge itself is still right.

Users give free signals too. A thumbs down, a request for a human, or the same customer returning within 24 hours with the same question all suggest a poor answer. Track these as rates on the dashboard.

How big a change can 800 samples show? At a 90% pass rate, one day's figure has a margin of error of about plus or minus 2.1 percentage points. On about 95 days out of 100, the day's figure lands within 2.1 points of the true rate.

To tell two days apart, the gap needs to be about 3 points. A one-point dip is noise; a jump in failures from 1% to 9% is not.

On v15, the price check failed on 9% of plan answers, up from 1% on v14. The judge's pass rate barely moved. Without the code check and the support team's reviews, the quality drop would have stayed silent.

Split the change into its parts, compare before and after by version, then read a few traces from the worst slice. Look in four places: your prompt or code, the model, your data, and the questions users ask.

For cost, daily cost equals conversations, times model calls per conversation, times tokens per call, times the price, with input and output tokens priced separately. Check each part in turn.

Pause and predict: traffic is flat, calls per conversation are flat, and the service reports the same model name. Which part moved?

Quality needs the same slicing. The engineer split the failed samples by topic. Bill questions were fine, but plan questions failed far more often. He then opened 20 failing traces and read the prompts and replies in the locked text store.

The cause was plain once he looked. The pasted plan list was an old copy from last quarter, so the model quoted old prices with confidence. The step-by-step line made replies about half as long again, and more customers gave up and asked for a person.

The team rolled back to v14 within the hour. For v16, the plan list came from the live plan database through a tool, and the step-by-step line was gone. They sent v16 to 10% of chats first and compared both versions on the dashboard before a full release.

Start with why error-rate monitoring is not enough: AI systems fail by giving worse answers or costing more while every request succeeds. Then walk through what you record, how you watch it and how you find causes.

1. 1

   Name the silent failures

   Say that a success code does not mean a correct answer, and that cost can drift with no error at all.

2. 2

   Define the per-call record

   List tokens, time, cost, tool calls, finish reason, and the prompt and model versions, both requested and reported.

3. 3

   Add traces for multi-step answers

   One root span per question and a child span per model call and tool, named with the OpenTelemetry generative AI conventions.

4. 4

   Protect personal data

   Redact before writing, keep text in a separate locked store with a short life, and record only references on spans.

5. 5

   Set dashboards and alerts

   Track cost per conversation, tokens, latency, tool errors and quality rates, split by version, with alerts on change from normal.

6. 6

   Score live samples

   Check a small daily sample with code, a judge model and people, and size the sample for the change you need to see.

7. 7

   Explain how you find causes

   Split cost into its parts, compare by version and topic, then read real traces from the worst slice.