Coding Agent Architecture: Harness, Tools and Sandboxes

It guesses, because a model on its own cannot open your files, run your tests or see what its change broke. Coding agent architecture is the code you build around the model so it can do those things safely.

Picture a common setup. A billing team keeps its Python service in one repository of about 1,800 files and 210,000 lines. The setup is made up, but the problem is real.

Ticket BILL-412 arrives. When a customer uses a discount code, the invoice charges GST, India's goods and services tax, on the full price. An item costs Rs 1,000 with a Rs 100 discount, so 18 percent GST should be Rs 162, but the invoice shows Rs 180.

An engineer fixes it by hand. A search for gst gives 23 matches, and in invoice_totals.py the tax is worked out before the discount. Swapping the order, adding a test that expects Rs 162 and running all 412 tests takes about 45 minutes (made up).

The team sees about 30 small tickets like this a week, so they want an agent that drafts each fix for an engineer to review. The first try pastes the ticket into a chat window. The confident answer edits tax_utils.py, a file that does not exist.

The model could not see the repository, open a file or run a test. Everything in this guide is about giving it those abilities with limits.

Which words do you need first?

You need six words. Each gets a plain meaning and an example from the billing bug.

Model. The model is the AI itself: a program that reads text and writes text back, often called a large language model (LLM). It cannot open files or run code on its own.

Harness. The harness is all the normal code around the model. It runs the loop, decides what the model sees, carries out its requests and stops the run. The Python program the team is about to write is its harness.

Tool. A tool is one action the model may ask for, like read_file or run_tests. The model writes the tool's name and inputs, and the harness runs it and returns the result. Example: search_code for gst returns 23 lines.

Sandbox. A sandbox is a closed-off place where the agent's commands run, so mistakes cannot reach real systems. Example: a throwaway copy of the repository inside a container, a lightweight box that limits what a program can see.

Context window. The context window is the most text the model can read in one request. It is measured in tokens, small pieces of text about the size of a short word. The billing repository is far too big to fit.

Patch. A patch, also called a diff, is a short record of exactly which lines changed. The agent's output for BILL-412 is a 9-line patch. An engineer reviews it like a pull request, a request to merge a change after review.

Coding agent architecture is the set of parts around the model that let it change real code safely: a loop, a context builder, tools, a sandbox and a test runner. The model sits in the middle, but most of the design work happens around it.

Think of a skilled mason building a wall. The mason is the model: the skill and the judgement. The scaffolding is the loop and the sandbox, a safe place to stand and work.

The sharp tools are the agent's tools. The plumb line, a weighted string that shows a true vertical, is the test suite. The site supervisor is the permission check and the engineer who reviews the patch.

Where the picture breaks: a mason remembers yesterday's work. The model remembers nothing between requests, so the harness must put everything it needs into each prompt, the text sent to the model in one request.

Here is the team's design. Start at the ticket and follow the arrows.

The coding agent loop works in rounds: the model asks for one tool, the harness runs it in the sandbox, and the result is added to the notes. It repeats until the model says it is done and the tests pass, or a round limit is reached.

Here are the rounds for BILL-412. Pause and predict: which round is the strongest proof that the fix is real?

1. 1

   Round 1: search

   The model asks for search_code with the text gst. It gets 23 matching lines from 6 files.

2. 2

   Round 2: read

   It reads invoice_totals.py, lines 40 to 140, and sees compute_gst run before apply_discount.

3. 3

   Round 3: write a test

   It adds test_gst_after_discount, which expects Rs 162 of GST on the Rs 1,000 item with a Rs 100 discount.

4. 4

   Round 4: prove the bug

   It runs only the new test. The test fails with 180, so the bug is real.

5. 5

   Round 5: fix

   It edits invoice_totals.py so GST is worked out after the discount.

6. 6

   Round 6: check the fix

   The new test runs again and passes.

7. 7

   Round 7: check everything

   It runs the full test suite, every test in the project. All 413 pass: the 412 old ones and the new one.

8. 8

   Round 8: done

   The model says it is done. The harness runs the tests once more itself, then builds the patch.

Round 4 is the answer. A test that fails before a change and passes after it shows the change fixed this bug, not just that nothing crashed.

You can check the arithmetic in the test: 18 percent of Rs 1,000 is Rs 180, and 18 percent of Rs 900 is Rs 162. With a small test like this, pytest, a Python test runner, prints 1 failed before the fix and 1 passed after it.

Here is the loop in Python, kept small. The helper functions are placeholders: ask_model sends the notes and tool list to the model, and the others do what their names say. TOOLS maps each tool name to a function that runs inside the sandbox. If you do not read Python, skip to the paragraph after the code.

def fix_ticket(ticket, max_rounds=30):
    notes = [build_start_context(ticket)]        # rules, repo map, ticket
    for round_number in range(max_rounds):
        reply = ask_model(notes, TOOLS)
        if reply.kind == "done":
            result = run_tests()                 # the harness checks, not the model
            if result.passed:
                return make_patch()              # a diff for a person to review
            notes.append("Tests failed: " + result.summary)
            continue
        if reply.tool not in TOOLS:
            notes.append(f"Error: there is no tool called {reply.tool}")
            continue
        try:
            output = TOOLS[reply.tool](**reply.args)   # runs inside the sandbox
        except Exception as err:                       # bad inputs, timeouts
            output = f"Error: {reply.tool} failed: {err}"
        notes.append(shorten(output, max_lines=60))
    return hand_to_person(ticket, notes)

What just happened? The limit of 30 rounds stops a confused agent from running all night. When the model says done, the harness runs the tests itself instead of trusting the claim.

Bad requests turn into error notes, not crashes. We ran this loop with a stand-in model on Python 3.13. Without the try block, a tool called with a wrong input name stopped the whole run with a TypeError.

A coding agent needs four kinds of tools: read, search, edit and run. A small, sharp set is easier for the model to use well, because every extra tool is one more way to pick wrong.

Notice what is missing: a shell, a tool that runs any command you type. OWASP, a non-profit that publishes security guidance, says in its Top 10 for LLM applications to avoid open-ended tools such as shell commands where possible. So the team offers run_tests and a short list of allowed commands.

Which edit format should the agent use?

For most changes, search and replace is the safest edit format, because the harness can check each edit before saving. The edit format is how the model describes a change. There are three common choices:

• Whole file: the model writes out the entire new file. It is simple, but slow for big files, and the model can quietly drop lines.
• Line range: the model says replace lines 52 to 55 with new text. It is short, but line numbers shift after every edit.
• Search and replace: the model quotes an exact old snippet and its new version. The harness refuses the edit if the old snippet is missing or appears twice.

Here is a search and replace tool in plain Python; we ran it on Python 3.11, 3.12 and 3.13. It refuses paths outside the project, unclear snippets and edits that would leave a Python file unreadable.

import ast
from pathlib import Path

WORKSPACE = Path("workspace").resolve()

def edit_file(path, old_text, new_text):
    target = (WORKSPACE / path).resolve()      # follows ../ and links
    if not target.is_relative_to(WORKSPACE):
        return "Error: you can only edit files inside the project folder."
    if not target.is_file():
        return f"Error: there is no file called {path}."
    source = target.read_text()
    matches = source.count(old_text)
    if matches != 1:
        return f"Error: old_text found {matches} times. Copy more lines so it is unique."
    new_source = source.replace(old_text, new_text)
    if target.suffix == ".py":
        try:
            ast.parse(new_source)               # syntax check only
        except SyntaxError as err:
            return f"Error: edit NOT applied. Line {err.lineno}: {err.msg}"
    target.write_text(new_source)
    return f"Edited {path}"

WORKSPACE.mkdir(exist_ok=True)
(WORKSPACE / "totals.py").write_text("tax = 0\nfee = 0\n")
print(edit_file("totals.py", "= 0", "= 1"))
print(edit_file("totals.py", "tax = 0", "tax = (0"))
print(edit_file("../secrets.txt", "x", "y"))
print(edit_file("total.py", "tax = 0", "tax = 18"))
print(edit_file("totals.py", "tax = 0", "tax = 18"))

# Output on Python 3.11 to 3.13:
# Error: old_text found 2 times. Copy more lines so it is unique.
# Error: edit NOT applied. Line 1: '(' was never closed
# Error: you can only edit files inside the project folder.
# Error: there is no file called total.py.
# Edited totals.py

What just happened? The first edit was refused because = 0 appears twice, and the second because it broke the file. The count check is the core: zero matches means the model misremembered the code, and two or more means the edit is unclear.

The path check uses resolve(), which follows ../ and links before comparing. In our test, a link inside the workspace that pointed at a system file was refused too.

A coding agent finds the right code by searching and reading small pieces, not by loading the whole repository. The harness keeps the context window for what matters now: the rules, the ticket and the latest few results.

A 2023 benchmark paper built 2,294 tasks from real issue reports on GitHub, a site that hosts code, in 12 popular Python projects. Its Table 1 reports those codebases average about 3,010 non-test files and 438,000 non-test lines.

More code is not more help. In that paper's simple retrieval setup, a keyword search called BM25, one model solved 1.96 percent of tasks when its context was capped at 13,000 tokens. At 50,000 tokens, with more retrieved code, the same model solved 1.22 percent.

Follow the diagram from the whole repository down to one focused prompt.

Here are the habits that make this work. The last three come from a 2024 paper on agent-computer interfaces:

• A repo map: a short outline of folders, files and function names, built by normal code. The model can guess where to look before reading anything.
• A project notes file: a small file that says how to run tests and what to avoid. The team's says never to edit database migration files, the scripts that change the database layout.
• Capped search results: at most 50 results; when there are more, the agent is told to search more narrowly.
• Small file windows: a 100-line window solved more tasks than a 30-line window or the whole file.
• Shortened old results: the last five results stay in full, and each older one shrinks to a single line.

When a long run still fills the window, the harness can compact the notes. The model summarises the older rounds, and that summary replaces them. The ticket and the rules always stay in full.

A coding agent checks its work by running real checks, mainly tests, not by asking the model whether it feels done. The harness runs those checks itself and passes a change to a person only when they succeed.

The benchmark paper scores fixes the same way. A task counts as solved only if two groups of tests pass. Fail-to-pass tests failed before the fix and must pass after it; pass-to-pass tests must keep passing, which shows nothing else broke.

The team's harness uses a ladder of checks, from the cheapest to the most expensive:

1. 1

   Syntax check on every edit

   The edit tool refuses any change that Python cannot parse, so broken files never reach the tests.

2. 2

   Reproduce the bug

   A new test must fail before the fix, like test_gst_after_discount expecting Rs 162.

3. 3

   Pass the new test

   After the fix, the same test must pass.

4. 4

   Run the full suite

   All 412 old tests must still pass, so nothing else broke.

5. 5

   Lint and type checks

   A linter flags likely mistakes, such as a name that is never defined. A type checker spots the wrong kind of value, like text where a number should be.

6. 6

   A person reviews the patch

   An engineer reads the 9 changed lines and the new test before anything is merged.

Tests only catch what they test. With no tests around discounts, the agent could pass every check and still be wrong. That is why a new failing test comes first, and why a person still reads the patch.

A coding agent needs a sandbox because it runs commands chosen by a model, and any of them could be wrong or harmful. The sandbox makes sure a bad command breaks only a throwaway copy, not your laptop, your secrets or production.

Honest mistakes are not the only risk. The agent also reads text written by others, such as packages (ready-made code libraries). A comment in one could hide an instruction to print every password.

OWASP calls this indirect prompt injection: the model takes in instructions from outside sources such as files or websites. Its 2025 guidance says it is unclear whether any prevention is foolproof. So you also limit what a fooled agent can do.

The team's sandbox follows five rules:

• A fresh copy of the repository for every ticket, thrown away after the run.
• No secrets: no production passwords, API keys (secret codes that let a program use a service) or customer data. Tests use fake data.
• Network off by default, except an internal package mirror, a local copy of approved packages.
• Time and size limits: every command stops after 5 minutes, with 2 CPU cores and 4 GB of memory.
• No push rights: the agent produces a patch, and only a person can merge it.

A plain container shares the kernel, the core of the operating system, with the host. If code finds a flaw in that shared kernel, it can reach the host. For code you do not trust, add a stronger wall.

One option is gVisor. Its docs describe an application kernel that intercepts a program's system calls (requests like open this file) and handles them itself, instead of passing them to the host kernel. Another option is a small virtual machine, a whole pretend computer with its own kernel.

Which actions need permission?

Actions that are hard to undo, or that reach outside the sandbox, need a person's yes. Reading and editing inside the workspace can run at once, because a patch is easy to throw away. Start at the top of the chart.

Often, yes. In a 2024 study that kept one model fixed, harness changes alone moved the share of tasks solved from 10.3 percent to 18.0 percent.

The study is the agent-computer interface paper. It tested on a 300-task subset of the 2023 benchmark and changed one harness feature at a time. The table shows the results from its Table 3.

Two lessons stand out. Without a proper edit command the agent solved about 23 fewer tasks. Showing the whole file did worse than a 100-line window, which fits the earlier finding that more code can hurt.

The same paper also compared harness and model directly, in its main results (Table 1). With one model, a simple retrieval setup with no loop solved 2.67 percent of the 300 tasks, and the agent harness 18.00 percent. A different model in the same harness solved 13.00 percent.

A second 2024 paper dropped the free-roaming loop entirely. It used three fixed phases: find the place to change, write candidate fixes, and check them with tests. Its revised version reported 32.00 percent (96 of the same 300 tasks) at low cost, the best of the open-source agents it compared.

That paper used a different model from the table's study, so its 32.00 percent and the 18.0 percent do not compare directly. What both show is that the design around the model moves the score.

The team saw the same pattern when it replayed 30 old tickets as a test set (made up). The first harness, with whole-file edits and no test loop, fixed 9. With search and replace edits, capped search and the check ladder, the same model fixed 19.

A newer, pricier model on the old harness fixed only 12 of the 30. A stronger model still helps. But a model swap is easy to try, while harness problems stay hidden until you read the runs.

A git command that catches weakened tests

To flag a patch that edits, deletes or renames an existing test, stage the agent's work with git add -A and compare it with the main branch. New test files do not show up, so the agent's own new test raises no false alarm.

# Run in the agent's copy of the repository (main = your base branch)
git add -A && git diff --cached --name-only --diff-filter=MDR main -- tests/

# Output on git 2.43, one line per touched test file:
#   fix plus a new tests/test_gst.py  -> (no output)
#   an existing test edited           -> tests/test_invoice.py
#   an existing test deleted          -> tests/test_refunds.py
#   an existing test renamed          -> tests/test_refunds_old.py (new name)

# Second check: new or changed conftest.py and pytest settings files
git add -A && git diff --cached --name-only main -- '*conftest.py' pytest.ini pyproject.toml setup.cfg tox.ini
#   a new tests/conftest.py that skips tests -> tests/conftest.py
#   a pytest.ini change that drops a test     -> pytest.ini
#   fix plus a new tests/test_gst.py          -> (no output)

The git add step matters. Without it, git diff never shows new untracked files. In the filter, M means modified, D deleted and R renamed.

The skipped-test trap

A passing exit code is not proof. In our test on pytest 9.1.1, a new tests/conftest.py, a pytest setup file, marked every test as skipped. The bug was still there, yet pytest printed 1 skipped and exited with code 0, the signal for success.

The first git command misses this, because conftest.py is a new file. It also misses a pytest.ini settings change that leaves out the failing test: in our test, pytest printed 2 passed, 1 deselected and exited with code 0. The second command catches both. Also treat any skipped test as a failure, and check that as many tests ran as before.

Six failures are worth designing for: the wrong file, edit loops, gaming the tests, invented code, getting lost in a long run and commands that never finish. Each one has a fix in the harness, not just in the prompt.

• Wrong file: the agent fixes old_tax.py, a copy nobody uses. Fix: a failing test first, since a change in the wrong file cannot make it pass.
• Edit loops: it sends the same broken edit five times. Fix: clear error notes, and stop after three identical failed actions.
• Gaming the tests: it changes the expected GST from 162 to 180. Fix: flag every change to an existing test for a person.
• Invented code: it calls a helper like round_paise that does not exist. Fix: a lint check for undefined names, then the tests.
• Lost in a long run: after 25 rounds, it forgets the discount part of the ticket. Fix: keep the ticket at the top of every prompt.
• Hanging commands: it starts a web server that never exits. Fix: a time limit on every command.

You can check the invented-code fix yourself. A file that calls round_paise still parses, so a syntax check passes it. Running flake8, a Python linter, with --select=F821 prints invoice_totals.py:3:12: F821 undefined name 'round_paise' and exits with code 1.

The team saves every run as a log, a record of each round. Most of its harness fixes came from reading failed runs, not from rewording the prompt.

Explain it as a loop around a model, then spend most of your time on the parts around it: tools, context, sandbox and checks. Walk through one concrete ticket to keep it real.

1. 1

   Pin down the job

   Say what the agent does and does not do. For example: draft fixes for small tickets in one repository, with a person merging every change.

2. 2

   Draw the loop

   Model asks, harness acts, result goes into the notes. Stop when the harness's own test run passes, or at a round limit.

3. 3

   Name the tools and edit format

   Read, search, edit and run tests, with search and replace edits and no general shell.

4. 4

   Handle a big repository

   A repo map, capped search, small file windows and shortened old output.

5. 5

   Add the sandbox and permissions

   A throwaway copy, no secrets, no network by default, and a person for risky actions.

6. 6

   Show how you verify

   A failing test first, then the full suite, lint and type checks, flags on weakened tests, then a human review.

7. 7

   Say how you would measure it

   Replay past tickets. Track how many get fixed, the cost per ticket and how often reviewers reject a patch.