Multi-Agent System Design: When Orchestration Is Worth the Cost

Costs rise, runs get harder to debug, and a fact can get lost each time work passes from one agent to another. Often nothing crashes; the final answer is simply wrong.

Multi-agent system design is deciding whether to split a job across agents, how to split it, and how the pieces share facts. This guide follows one job from start to finish.

Picture a common setup. An accounting software team prepares a month-end pack for about 400 small business clients in the first week of every month. The setup is made up, but the problem is real.

Each pack has a bank check, a GST summary, a look at expenses and two pages of plain notes. GST is India's goods and services tax.

Take one client, a wholesaler, for March. It has 2 bank accounts and 1,180 bank lines. The finance team matches each bank line to an entry in the books, the client's record of every sale and payment.

They find 3 lines that do not match, worth Rs 42,300 (Indian rupees) in total, and explain them in the notes. One pack takes about 45 minutes by hand, so 400 clients take 300 hours in one week.

The team's first AI version is one agent with five tools. It works for small clients. On big clients with 5,000 bank lines, it gets slow and sometimes misses an unmatched line.

A teammate proposes six agents: a manager agent that hands out work and five agents that each do one job. The finance lead asks two questions. Will six agents get the numbers right more often, and what will it cost?

You need six words. Each gets a plain meaning and an example from the month-end job. Our AI agent vs workflow guide covers the basics if the word agent still feels fuzzy.

Agent. An agent is a model working in a loop: it picks a tool, sees the result, and picks again until the goal is done. Example: the agent asks for the bank lines, then the books, then compares them.

Everything the agent has read so far, its instructions plus every tool result, is what this guide calls its notes. In a typical agent loop, the whole set of notes is sent to the model again on every round.

Multi-agent system. A multi-agent system is two or more agents, each with its own instructions, its own tools and its own notes, passing work between them. Example: a bank agent, a GST agent and a writer agent.

Orchestrator. The orchestrator is the part that splits the job, hands out the pieces, and collects the results. It can be a model, or it can be plain code.

Worker. A worker, often called a sub-agent, is an agent that does one piece and reports back to the orchestrator. Example: the bank worker matches the 1,180 bank lines and returns the 3 that do not match.

Handoff. A handoff is the moment work passes from one agent to another, along with whatever the first agent learned. Example: the bank worker passes "3 unmatched lines, Rs 42,300" to the writer.

Shared state. Shared state is the one place where all agents read and write the facts about a job. Example: a database table holding every number for the wholesaler's March pack.

It is worth the cost when the job splits into independent parts that each need their own notes, tools or instructions. Even then, keep it only if it beats one agent on real examples.

The reason is the context window: the limit on how much text a model can read in one request. Every round, the agent's instructions and all tool results so far must fit inside it.

For a small client, everything fits easily. For a big client, 5,000 bank lines plus GST and expense data make the notes very long. Fitting is not the same as being used well.

A 2023 study found that models used a fact in the middle of a long input less reliably than a fact at the start or end. That held even for models built for long inputs.

Splitting helps here. The bank worker reads all 5,000 lines in its own notes and hands back a short result. But the split adds more model requests, more handoffs, and more places where a number can go wrong.

Think of how the finance lead runs the human team. The lead is the orchestrator: they split the work and check every result. Each staff member is a worker with their own desk of papers, their notes. The office ledger is the shared state.

Where the picture breaks: a person who is unsure walks over and asks. A worker agent sees only what it was handed, and it often does not ask. So the checks a lead does by habit must be built in code.

In the orchestrator-worker pattern, one orchestrator splits the job into pieces, sends each piece to a worker, and joins the results into one answer. Workers report only to the orchestrator, not to each other.

Here is the teammate's six-agent sketch. It is the naive first try, shown so you can see what goes wrong. The workers hand their summaries straight to a writer, so it is part orchestrator-worker and part chain.

The orchestrator can be a model or plain code. Use a model when the split changes from job to job, as in a research task where the pieces depend on what the first search finds. Use code when the split is always the same.

The month-end split never changes. Every client always needs bank work, GST work and expense work. So the manager agent is not needed at all. Plain code can hand out the three pieces, with no model cost and no wrong plans.

How do parallel sub-agents save time?

Parallel sub-agents save time by running at once instead of one after another. The bank, GST and expense pieces do not need each other's results, so there is no reason to wait.

Using made-up numbers, say the bank worker takes 4 minutes, GST takes 3 and expenses take 3. One after another, that is 10 minutes. In parallel, the step takes as long as the slowest worker: 4 minutes.

What does not change is the bill. You still pay for every request each worker makes, so parallel work saves time, not money.

Here is a code orchestrator you can run on Python 3.13 with no installs. Each fake worker sleeps for a made-up time instead of calling a model, so you can watch the timing.

import time
from concurrent.futures import ThreadPoolExecutor

# Made-up times: each fake worker sleeps instead of calling a model.
WORK_SECONDS = {"bank": 0.4, "gst": 0.3, "expenses": 0.3}

def run_worker(name, task, broken):
    time.sleep(WORK_SECONDS[name])
    if name in broken:
        raise RuntimeError(f"{name} worker failed")
    return {"worker": name, "client": task["client"]}

def build_month_end(client, month, broken=()):
    task = {"client": client, "month": month}
    with ThreadPoolExecutor(max_workers=3) as pool:
        futures = {n: pool.submit(run_worker, n, task, broken)
                   for n in WORK_SECONDS}
    failed = [n for n, f in futures.items() if f.exception()]
    if failed:
        return f"send to a person: {failed}"
    return "save to run record, then run the writer"

for broken in [(), ("gst",)]:
    start = time.perf_counter()
    outcome = build_month_end("WHOLESALER-01", "2026-03", broken)
    print(f"{outcome} | {time.perf_counter() - start:.1f} s")

Expected output: "save to run record, then run the writer | 0.4 s", then "send to a person: ['gst'] | 0.4 s". The three workers need 1.0 s of work in total, yet each run takes 0.4 s, the slowest worker's time.

ThreadPoolExecutor, from Python's standard library, runs the workers at the same time. Leaving the with block waits for every worker. Then f.exception() returns any error a worker raised, or None if it finished without an error.

One trap for engineers: Python has no supported way to stop a running thread from outside. A low-level trick we tried could not interrupt a thread blocked in a sleep or socket read. In our run, f.result(timeout=0.5) raised TimeoutError, cancel() returned False, and the with block still waited the full 2 seconds.

So put the time limit inside each worker, for example a timeout on every model request and a cap on rounds. Only after all workers succeed does the code save their results to the run record.

During a handoff, one agent passes the work and a summary of what it learned to the next agent. Anything left out of that summary is lost, because the next agent cannot see the first agent's notes.

There are two common kinds of handoff. In call and return, the orchestrator calls a worker like a tool and gets a result back, so the orchestrator stays in charge. In hand over control, one agent passes the whole job to another and steps away.

The diagram below follows one fact through the naive sketch. Watch what happens to the detail at each step.

The bank agent knew the exact three lines. Its summary kept the count and the total but dropped which lines they were. The writer softened it further, so the reviewer had nothing it could check.

Where should shared state live in a multi-agent system?

Shared state should live in one run record that your code controls, not in messages between agents. Every worker writes its results into that record, and later agents read from it.

The team's run record follows five rules:

• Each worker writes only its own section. The bank worker cannot change GST numbers.
• Results are saved as fields, like unmatched_lines with the bank line IDs, not as a paragraph of text.
• The writer agent reads numbers from the record and may only quote them, never work them out again.
• Every entry saves which worker wrote it and when, so you can trace a wrong number back to its source.
• Code, not a model, decides when the record is complete enough to start writing.

Cost multiplies because every agent re-reads its own instructions and inputs on every round. Failures multiply because the final answer is right only if every agent and every handoff is right.

Start with cost. Model services typically charge by tokens, which are small chunks of text, often a short word or part of a word. You pay for tokens going in and tokens coming out.

Prompt caching, where the service stores the unchanged start of a prompt, makes repeat reading cheaper but not free. On one major provider's price list, reading from the cache costs at most a tenth of normal input, as of 19 October 2026.

Here is the team's measurement on the wholesaler's March pack. All numbers in the table are made up for this example.

Why more than twice the requests? The manager plans and reviews. Each worker reads its own instructions again. The writer and reviewer go back and forth once to fix a draft. None of that work exists with one agent.

Rs 15,200 a month is still small next to 300 hours of a team's time. The bigger cost is usually reliability, and the hours spent finding which agent broke a number.

The answer is about 83. If each agent's mistakes are independent of the others, the pack is right only when all six are right: 0.97 multiplied by itself six times, or 0.833.

You can check it with python3 -c "print(round(pow(0.97, 6), 3))", which prints 0.833. To keep the whole pack right 97 times in 100, each of the six agents would need about 99.5 right in 100.

Real mistakes are not always independent. A checker can catch some, and one bad input can break several agents at once. The lesson still holds: every extra agent must be clearly better at its piece just to break even.

One agent with more tools usually wins when the parts of the job depend on each other, or when all the notes still fit and are used well. Then splitting adds handoffs and cost without adding skill.

Signs that one agent is the better choice:

• The steps are a chain, where each step needs the last one's result, like reading the books before you can explain a mismatch.
• The full job fits in the agent's notes, and tests show it does not miss facts.
• The pieces need the same facts. Splitting them means copying the same data into every agent.
• You have not yet measured one agent on real examples. Without that baseline, you cannot show the team is better.

Often the fix is a better tool, not another agent. The single agent struggled with 5,000 bank lines because it read them all as raw text. So the team added a tool called match_bank_lines.

That tool does the exact matching in normal code and returns only the lines that did not match. The agent's notes shrank, and the missed-line problem went away for most clients.

At least one major model service lets one agent ask for several tools in a single response, as of 19 October 2026. That gives you some of the speed of parallel workers with only one agent.

Ask three questions in order: does one agent already do well, are the parts a chain, and is each part big or different? Only a no, then a no, then a yes leads to workers. A workflow, one of the chart's answers, is a fixed sequence of steps in your code.

Run the month-end job through it. For small clients, one agent with the new matching tool does well, so they stop at the first box. For big clients, bank, GST and expense work are independent and each is large.

So the team builds a mixed design. A size check in code sends each client down one of two paths. Both paths write into the same run record, and both end with a number check and the finance lead's approval.

Compare this with the naive sketch. The manager agent became code. The reviewer agent became a number check in code, which never gets tired. It flags numbers in the notes that are not in the record, and record numbers the notes leave out. Only the parts that need a model are still agents.

The cost stays close to one agent's. With made-up prices of Rs 14 for a small pack and Rs 30 for a big one, 360 small and 40 big clients cost Rs 6,240 a month.

Recent studies point the same way: more agents can help on work that splits into independent parts, and tend to hurt on step-by-step work. Treat each number below as one study's result, on its own tasks and models.

A study of 260 setups compared one agent with four multi-agent designs across six tasks, using the same tools, prompts and computing budget for each. With a central orchestrator, results rose 80.8% over one agent on a finance task that split into parts.

On a step-by-step planning task, every multi-agent design did worse than one agent, by 39% to 70%. The authors also report that once one agent already scored above about 45% on a task, adding agents tended to lower results.

The same study measured how much extra work coordination failures caused, read from the run traces, the saved logs of each run. On that scale one agent scores 1. Agents working alone, with their answers simply combined, scored 17.2. With a central orchestrator checking results, the score fell to 4.4.

That scale is not the share of wrong answers. Counted as failed tasks, the authors give a much smaller gap: about 1.1 to 1.3 times one agent's error rate.

It also found that tool-heavy tasks tend to pay the most for coordination. The authors' reading is that splitting the work splits each agent's token budget, leaving too little room to handle many tools.

A separate study gave one agent and a team the same thinking budget, meaning the same number of tokens the model may write while working a problem out. On multi-hop questions, which need several facts joined in a chain, the single agent matched or beat the team.

Those authors argue that many reported team gains come from spending more computation or from context effects, not from the split itself.

A third study found that the advantage of multiple agents shrinks as models get better. Its authors suggest a hybrid that passes requests between a single agent and a team, much like the size check above.

Finally, a study of more than 1,600 annotated runs, across seven popular multi-agent frameworks, sorted the failures into three families:

• System design problems: agents that ignore their task or role, repeat steps, lose the conversation history, or do not know when to stop.
• Agents not lining up: one agent ignores or withholds information, drifts off the task, or fails to ask for clarification, like the bank summary above.
• Weak checking: the run stops too early, or nobody verifies the answer properly, so a wrong pack is approved anyway.

You test a multi-agent system in layers: each worker alone, then each handoff, then whole runs on examples where you already know the right answer. You check both the final report and the trace.

A trace is the saved record of every model request, tool call and handoff in one run. The team asks the finance lead for 20 golden months: past packs the finance team already checked by hand, so the right numbers are known.

1. 1

   Test each worker alone

   Give the bank worker one golden month's bank lines and check it finds exactly the known unmatched lines. Do the same for GST and expenses.

2. 2

   Test the handoffs

   Check that every worker's result has the right fields and types before it reaches the run record. A missing field should fail the test, not slip through.

3. 3

   Run whole packs on golden months

   Compare every number in the final pack with the finance team's answer. Numbers must match exactly; the wording of the notes can differ.

4. 4

   Read the trace, not just the answer

   Check which workers ran, how many requests were made, and what the run cost. A right answer that took 200 requests is still a bug.

5. 5

   Break things on purpose

   Make one worker time out or return nonsense. The pack must go to the finance lead, never to the client half-built.

6. 6

   Run each case several times

   Model answers can vary between runs, so one pass proves little. Run each golden month a few times and count how often it is fully right.

7. 7

   Compare with one agent

   Run the same golden months through the single-agent version. Keep the multi-agent path only where it clearly wins.

The number check from the final design is small enough to show in full. It lists numbers in the writer's notes that are not in the run record, then record numbers the notes leave out. It runs on Python 3.13 with no installs.

import re

run_record = {"unmatched_count": 3, "unmatched_total": 42300,
              "gst_payable": 18450}

def numbers_in(text):
    found = re.findall(r"\d[\d,]*", text)
    return {int(n.replace(",", "")) for n in found}

def unknown_numbers(notes, record):
    return sorted(numbers_in(notes) - set(record.values()))

def missing_numbers(notes, record):
    return sorted(set(record.values()) - numbers_in(notes))

good = "3 bank lines worth Rs 42,300 are unmatched. GST due is Rs 18,450."
bad = "3 bank lines worth Rs 42,800 are unmatched. GST due is Rs 18,450."
vague = "Some payments are still being matched."
for notes in [good, bad, vague]:
    print(unknown_numbers(notes, run_record), missing_numbers(notes, run_record))

Expected output: [] [] for the good notes, then [42800] [42300] for the bad notes, then [] [3, 18450, 42300] for the vague notes. The bad notes changed one digit of the unmatched total. The vague notes have no numbers to catch, so only the second list flags them.

A real check needs more rules than this sketch. If two record numbers swap places, both lists stay empty, so it must also match each number to its field. Dates such as 2026-03, percentages and rounded amounts all produce numbers, so decide how each one is allowed to appear.

On the 20 golden months, using made-up results, the single agent with the matching tool got 18 fully right. The six-agent sketch got 16 right at about 2.7 times the cost. The mixed design got 19 right.

That answers the finance lead's two questions with evidence, not opinion. Every time the lead catches a mistake in a real pack, that month becomes a new golden month.

The strong answer starts with one agent and adds more only where you can name the reason and the cost. Interviewers want to see that you treat every extra agent as a cost you must justify.

1. 1

   Start with one agent

   Describe one agent with a short list of good tools, and say how you would measure it on real examples.

2. 2

   Name the reason to split

   Point to independent parts that would each flood one agent's notes, like 5,000 bank lines, or need their own tools. If you cannot name one, do not split.

3. 3

   Pick the orchestrator

   Say whether the split is fixed, so code can orchestrate, or changes per job, so a model must plan it.

4. 4

   Design handoffs and shared state

   Explain that workers write structured results to one run record, and that code copies numbers, never the model.

5. 5

   Count cost and failure

   Estimate requests per job. Show how a 3 in 100 error rate per agent becomes about 17 in 100 across six agents, if errors are independent.

6. 6

   Explain testing and limits

   Describe golden examples, trace checks, a cost budget per run, and least-privilege tools, meaning each worker gets only the tools its piece needs. Add a person approving anything risky.