How to Evaluate AI Agents: Trajectories, Not Just Answers

An AI agent can give a correct-looking answer while doing something wrong on the way, like placing the same order twice. That is why learning how to evaluate AI agents starts with what the agent did, not what it said.

Picture a common setup. A grocery shop in Varanasi uses an AI agent to track stock and order from suppliers. The owner types requests in a chat window, and a two-person team built and tests the agent. The setup is made up, but the problem is real.

One evening the owner types a short message. "Basmati rice is down to 6 bags. Order 40 more from our usual supplier." A few seconds later the agent replies. "Done. 40 bags ordered for Rs 28,000."

Two days later a van brings 80 bags, not 40, with a bill for Rs 56,000. The shop has no room for the extra bags, and its cash is now stuck in rice.

The team had tested the agent before launch. They wrote 30 sample requests and read each final reply. All 30 replies looked right, including a rice order just like this one.

So they open the record of that evening and read it step by step. The agent placed the order, but the supplier's system was slow and sent no answer in time. The agent ordered again without checking, and the first order had gone through after all.

The reply was right. The path to it was wrong. The rest of this guide changes the team's tests so this mistake fails in testing, before the shop ever sees it.

To evaluate AI agents, check three layers of every run: what the agent said, what is really true at the end, and how it got there. The second and third layers catch the costly mistakes.

First, what is an agent? An AI agent is a program where an AI model picks its own next step, again and again, until the job is done. The model is the kind of text-writing AI behind chat apps.

It acts by calling tools. A tool is a small function the agent is allowed to run, such as check_stock or place_order. The values it passes to a tool, like "item: rice, qty: 40", are called arguments.

One picture carries the whole guide: a Class 10 maths board exam. If you studied in India, you know how it is marked.

• Student = the agent. It decides every step on its own.
• Exam question = the task, for example "order 40 bags of rice".
• Answer in the box = the agent's final reply.
• The working on the answer sheet = the trajectory, the full record of every step the agent took.
• Examiner with a marking scheme = the grader (whatever gives the score), using a rubric (a written list of what earns marks).
• Mock tests before the board exam = test runs in a fake copy of the shop.

The team's 30 tests read only the first layer. The rice bug lived in the other two.

You have probably met this problem as a customer. A help chat says your refund is done, but the payment tool had failed and the agent never checked. A travel app books a flight when you only wanted to look, because it skipped asking first.

An agent trajectory is the full record of one run: every message, every tool the agent called, the arguments it passed and the result it got back. Some teams call it a trace or a transcript.

Here is the rice run from that evening, one box per step.

Why did the agent order twice? A timeout means the tool gave no answer in time, which does not mean the order failed. The agent read silence as failure and tried again. A careful agent checks open orders first.

Grade each step with four questions: right tool, right arguments, no wasted steps and nothing unsafe. Each question is one line on the examiner's marking scheme.

• Right tool: for a rice order, the agent should use check_stock and place_order, never change_price. The wrong tool can do real damage.
• Right arguments: 40 bags of rice from the usual supplier is right; 4 bags, or another supplier, is wrong. A right tool with wrong values is still a wrong order.
• No wasted steps: calling check_stock three times for the same item adds cost and waiting, and every step costs the shop time and money.
• Nothing unsafe: a duplicate order, a deleted item or an order above the owner's limit fails the run at once. Some mistakes cost more than every good run earns.

Should the agent follow one exact path? No. A maths examiner gives marks for any correct method, not just the one in the answer key. Do the same for agents.

Write rules that every good path must follow instead: check stock before ordering, place exactly one order, never change prices. Any path that keeps the rules and reaches the right end state passes.

A worked example: two good runs, two different paths

The path the team expected is: check stock, check open orders, place the order, reply. Run A follows it exactly. Run B checks open orders first, then stock, then places the order and replies.

Use plain code wherever code can decide, an AI judge where judgement is needed, and people to check that the judge is fair. Each grader has a job it does best.

Code is cheap, fast and gives the same result every time. So it handles counts, values and the end state. Counting place_order calls is a job for code.

An AI judge is an AI model that reads the trajectory and scores it against a rubric. People often call this LLM-as-a-judge, where LLM (large language model) means the kind of AI model behind chat apps.

The team's rubric for replies has four questions, each answered yes or no:

• Does the reply name the item, quantity, supplier and total price?
• Does it match what the tools really did, with nothing made up?
• If something failed, like a supplier timeout, does it say so plainly?
• Is it short and polite enough for a busy shop owner?

On the bad rice run, question 2 fails: the reply says one order of 40 bags, but the trajectory shows two. Narrow yes-or-no questions give answers you can count. "Was this good?" gives vague scores.

Can you trust the judge? Not blindly. In a 2023 study by Zheng and others, AI judges compared pairs of chat answers. Leaving out ties, the strongest judge agreed with human judges over 80% of the time, about as often as humans agreed with each other.

The same study found biases too. Judges tended to favour the answer shown first and favour longer answers. Some also seemed to favour their own answers, though the authors could not confirm it. That study judged chat answers, not agent steps, so test your own judge.

So the team marked 30 runs by hand, then ran the judge on the same 30. Where they disagreed, they rewrote that rubric question and checked again. It is the examiners' moderation meeting, done for an AI judge.

An agent can pass once and fail the next time because it does not always take the same path, even on the same task. So you run every task several times and count how often it passes.

Why does the path change? Models usually pick each next word with some randomness, and one different word can lead to a different tool call. Tools change too, like a supplier that is slow tonight and fast tomorrow.

The team turns the rice bug into task 17 and runs it 5 times. In 4 runs, the agent checks open orders after the timeout and places one order. In 1 run, it orders twice.

1. 1

   One run

   The task passes 4 times out of 5, so the chance of a good run is about 0.8, or 8 in 10.

2. 2

   A busy week

   The owner sends about 20 order messages a week. If every order behaved like task 17, 2 bad runs in every 10 means about 4 bad orders a week.

3. 3

   Five runs in a row

   If runs are independent, the chance that all 5 pass is 0.8 multiplied by itself 5 times. That is about 0.33, roughly 1 in 3.

A 2024 research benchmark for customer-service agents, by Yao and others, introduced pass^k. With the agents tested in June 2024, even state-of-the-art ones solved under half the tasks. For the agent it highlighted, pass^8 in retail tasks fell below 25%.

Build it from realistic tasks, each with a starting state, a request and a clear goal. Then run every task in a fake copy of your systems, so the agent can act freely without touching real stock or money.

The team started with the owner's real messages from the first month, and turned every bug into a task. Their set now has 40 tasks of five kinds:

• Everyday tasks: a normal order, a stock check, a question about a price.
• Tricky tasks: two kinds of rice are in stock, so the agent must ask which one the owner means.
• Failure tasks: the fake supplier times out or runs out of stock, on purpose.
• Safety tasks: a request for 500 bags worth Rs 3.5 lakh (350,000 rupees), which must wait for the owner's approval.
• Do-nothing tasks: stock is already high, so the right move is to reply without ordering.

The fake shop is a small stock database and a pretend supplier, with the same tools as the real ones. Every run starts from a fresh copy, so leftovers from one run cannot help or hurt the next.

Success is judged mainly by the final state of that fake shop. The 2024 benchmark works the same way: it compares the database at the end of each chat with the expected one. It also checks that the agent told the user the facts the task needed.

For tasks that need a back-and-forth chat, another AI model plays the owner. It knows which rice the owner wants, but only says so when asked. The 2024 benchmark also used a language model to play its customers.

This is not only a beginner's worry. A 2024 paper, AI Agents That Matter by Kapoor and others, found many agent benchmarks had weak holdout sets or none, so agents overfit to the test.

Run each task n times and count the c runs that passed. pass^k is the number of ways to pick k runs from the c good ones, divided by the ways to pick k from all n. Average that over tasks.

That is the pass^k estimate from the 2024 benchmark paper. pass@k uses the matching formula from a 2021 paper on code-writing models. It is one minus the ways to pick k from the failed runs, over the ways to pick k from all.

The team reruns task 17 ten times, and 8 runs pass. This code prints both measures and the quick shortcut, (c/n) raised to the power k. It needs Python 3.8 or later for math.comb.

from math import comb

def pass_at_k(n, c, k):
    return 1 - comb(n - c, k) / comb(n, k)

def pass_hat_k(n, c, k):
    return comb(c, k) / comb(n, k)

n, c = 10, 8  # task 17: 10 runs, 8 passed
for k in (1, 3, 5):
    print(f"k={k}  pass@k={pass_at_k(n, c, k):.3f}  "
          f"pass^k={pass_hat_k(n, c, k):.3f}  (c/n)^k={(c / n) ** k:.3f}")

Running it with python3 should print exactly these three lines:

k=1  pass@k=0.800  pass^k=0.800  (c/n)^k=0.800
k=3  pass@k=1.000  pass^k=0.467  (c/n)^k=0.512
k=5  pass@k=1.000  pass^k=0.222  (c/n)^k=0.328

Why not use the shortcut? Raising a measured rate to a power reads too high on average. We checked it exactly: with a true pass rate of 0.8, 10 runs and k = 5, the true answer is 0.33.

Averaged over every possible outcome of those 10 runs, the formula gives 0.33 too. The shortcut averages 0.41. The gap grows when you have fewer runs, and at 5 runs it averages 0.48.

Two limits to know. You need at least k runs per task; with fewer, comb returns zero and the code divides by zero. The formula also assumes runs are independent, so reset the fake shop before every run.

Give each kind of task a step limit and a cost limit, then fail any run that goes over, even if its result is right. Base the limits on what a good run of that task usually needs.

Every step costs time and money. Each step is usually one more call to the model, and model services usually charge by tokens (small pieces of text, often a word or part of one).

If your agent sends the whole record so far with every call, as simple agent loops do, later steps send more text than early ones. So a loop that repeats steps costs more with each turn.

A good rice order takes 4 to 5 steps and costs about Rs 1.50 in model fees (made up). The team sets the limit at 8 steps and Rs 4 per task. That leaves room for one sensible extra check, but not for a loop.

Report cost next to the pass rate. The 2024 paper by Kapoor and others makes the same case: judging agents on accuracy alone has produced agents that are needlessly complex and costly.

Run a small set of tasks on every proposed change and the full set every night. Block any change that breaks a safety rule or drops the pass rate by more than chance would.

Teams call this CI (continuous integration): tests that run by themselves whenever someone proposes a code change. A tiny prompt edit or a renamed tool can change which tools the agent picks. Nothing crashes, so normal tests still pass.

Here is the step check the team's CI runs on every trajectory. A trajectory is a list of Step records, each with a tool name and its arguments. The last lines run it on the bad rice run and on Run B.

from dataclasses import dataclass, field

@dataclass
class Step:
    tool: str
    args: dict = field(default_factory=dict)

@dataclass
class Task:
    expected_orders: int
    expected_qty: int
    max_steps: int

FORBIDDEN = {"change_price", "delete_item"}

def check_steps(steps, task):
    problems = []
    tools_used = [s.tool for s in steps]
    if "check_stock" not in tools_used:
        problems.append("never checked stock")
    orders = [s for s in steps if s.tool == "place_order"]
    if len(orders) != task.expected_orders:
        problems.append(f"placed {len(orders)} orders, expected {task.expected_orders}")
    for s in orders:
        if s.args.get("qty") != task.expected_qty:
            problems.append(f"wrong quantity: {s.args.get('qty')}")
    if any(s.tool in FORBIDDEN for s in steps):
        problems.append("used a forbidden tool")
    if len(steps) > task.max_steps:
        problems.append(f"took {len(steps)} steps, budget is {task.max_steps}")
    return problems

task = Task(expected_orders=1, expected_qty=40, max_steps=8)
bad_run = [Step("check_stock", {"item": "rice"}),
           Step("place_order", {"item": "rice", "qty": 40}),
           Step("place_order", {"item": "rice", "qty": 40})]
run_b = [Step("check_open_orders", {"item": "rice"}),
         Step("check_stock", {"item": "rice"}),
         Step("place_order", {"item": "rice", "qty": 40})]
print("bad run:", check_steps(bad_run, task))
print("run B:", check_steps(run_b, task))

Running it with python3 should print these two lines:

bad run: ['placed 2 orders, expected 1']
run B: []

Each change's scores are compared with a baseline, the scores of the last version the team accepted. Follow one change through the gate below.

How big a drop is just luck?

Pass rates move by chance, so set the blocking line with numbers. Ten tasks run 3 times each give 30 runs. Say the accepted version passes 90% of the time.

Treat each run as an independent coin toss with a 90% chance. Then a typical swing (one standard deviation) is about 1.6 runs, or 5.5 percentage points, between checks. This code shows how often an unchanged agent scores low by luck alone.

from math import comb

def chance_at_or_below(score, runs, p):
    return sum(comb(runs, i) * p**i * (1 - p)**(runs - i)
               for i in range(score + 1))

for score in (26, 25, 24, 23):
    print(f"{score}/30 or lower: {chance_at_or_below(score, 30, 0.9):.3f}")

Running it should print these four lines:

26/30 or lower: 0.353
25/30 or lower: 0.175
24/30 or lower: 0.073
23/30 or lower: 0.026

An unchanged agent scores 26 or lower about 35 times in 100, so blocking there would stop good changes all the time. At 23 or lower it happens about 3 times in 100, so the team blocks there.

Real runs are not independent coin tosses: hard tasks tend to fail together. So also measure your own swing by running the unchanged version a few times, and widen the line if it moves more.

Explain it as layers, and tie each layer to one example task. Walk through these six points, one line each.

1. 1

   Define success as an end state

   Each task has a goal state, like exactly one order of 40 bags, checked in a fake environment.

2. 2

   Check the steps

   Right tool, right arguments, no wasted steps, nothing unsafe, written as rules rather than one fixed path.

3. 3

   Set budgets

   Give each kind of task a step limit and a cost limit, and report cost next to the pass rate.

4. 4

   Build the test set

   Mix everyday, tricky, failure, safety and do-nothing tasks, and keep some hidden.

5. 5

   Pick graders by cost

   Code first, an AI judge with a yes-or-no rubric next, and hand-marked runs to check the judge.

6. 6

   Repeat and gate

   Run each task several times, report pass^k, and block changes that break safety or drop more than luck allows.