Architecture Decision Records for AI Systems, With Examples

The reasons disappear, and a later change quietly undoes them. Decisions get made in meetings and chat threads, and nobody keeps AI architecture decision records, short files that say what was decided and why. The code shows what the system does today. It rarely says why.

Picture a common setup. An online marketplace for small sellers runs an AI assistant inside its seller app. Sellers ask it about the marketplace's rules, such as the commission on a sari over Rs 5,000 or how many days they have to ship. The setup is made up, but the problem is real.

Say about 6,000 sellers send around 1,800 questions a day. In April, a new engineer asks a fair question: why not fine-tune the model on the policy documents? Fine-tuning means training a model further on your own examples.

The tech lead remembers a long meeting about exactly this. Nobody remembers what was decided, or why. Two days of scrolling through old chat messages recover about half the reasons.

Then a worse problem surfaces. In March, someone switched the assistant to a cheaper model to save money. Wrong answers about return rules rose from about 3 in 100 to 14 in 100, and nobody noticed for 11 days. Nobody had written down what score counted as good enough.

The fix is not a new tool. It is a habit: every important decision gets a short written record, kept next to the code. The rest of this guide follows the team as it starts writing them.

You need six words to follow this guide. Each gets a plain meaning and an example from the marketplace assistant.

Architecture decision. An architecture decision is a choice that changes how the system is built or how well it behaves, and is costly to undo. Example: the assistant finds answers by searching the policy documents, not by retraining the model. Renaming a variable is not one.

ADR. An ADR (architecture decision record) is a short document that captures one architecture decision and the reasons behind it. Example: a file called 0007-retrieval-over-fine-tuning.md in the team's code repository, the shared folder of code kept under version control, usually with Git.

Decision log. The decision log is the whole collection of ADRs for a project, numbered in order. Example: the team's docs/decisions folder, holding ADR-0001 to ADR-0016.

Status. The status says where a decision stands today. The original 2011 list is proposed, accepted, deprecated (no longer applies) and superseded. The MADR template, explained below, adds rejected. Example: "Accepted, 2026-04-18".

Eval set and threshold. An eval set (short for evaluation set) is a fixed list of test questions with known correct answers that you run after every change. The threshold is the lowest score you agreed to accept. Example: 300 seller questions, and at least 90 in every 100 answered correctly.

Supersede. To supersede a decision means to replace it with a newer one while keeping the old record. Example: "ADR-0011 is superseded by ADR-0016". The old file stays, with a status line pointing to the new one.

An architecture decision record is a one or two page document about a single important decision, the situation that led to it, and its consequences. AI architecture decision records are the same files, aimed at the choices that shape how an AI system behaves.

The format was popularised by a blog post Michael Nygard published on 15 November 2011. He proposed small numbered files in the project repository, under doc/arch/adr-NNN.md, with numbers never reused. His reason: large documents are never kept up to date, and nobody reads them.

Think of the ledger in a kirana shop, a neighbourhood grocery store. The shopkeeper writes each entry with a date and never rubs one out. If a customer pays back a loan, a new line goes below; the old line stays. A decision log works the same way: a changed decision gets a new record, and the old one stays readable.

Where the picture breaks: a ledger line never changes at all. An ADR's status line does change, for example from Accepted to Superseded. Only the decision and its reasons stay frozen.

What sections does an ADR have?

Nygard's original had five sections: title, context, decision, status and consequences. MADR adds the options you considered and an optional Confirmation part that says how compliance will be checked. The table shows each section, filled in for the team's ADR-0007.

Nygard asks for every consequence, not just the positive ones, and this is the section people skip. Every real decision has a downside. An ADR that lists only good things is an advertisement, not a record.

Their key decisions change often, hide in places that do not look like code, and are judged by numbers someone chose. Without a record, the next person cannot tell a careful choice from an accident.

In ordinary software, a database choice might stay the same for years. In an AI system, the model can change under you. In the made-up setup, the team's model provider has also announced a retirement date for the model version in use. Another switch is coming, and without a recorded pass mark it could repeat the March incident.

A 2022 study of 174 Stack Overflow posts, plus 128 GitHub issues from 32 AI-based projects, looked at which architecture decisions AI developers discuss. Technology, component and data decisions were the most common types it found. These are the AI decisions that now get an ADR in the team:

• Model choice: which model answers which kind of question, and what happens when that version is retired.
• Prompt design: the instructions sent with every question. A prompt looks like an ordinary string in code, but it decides how the assistant behaves.
• Eval thresholds: what score on which eval set counts as good enough to release.
• Data rules: which documents the assistant may search, what seller data leaves the company, and how long chat logs are kept.
• Retrieval or fine-tuning: how the assistant learns the marketplace's rules in the first place.
• Autonomy: which actions the assistant may take alone, and which need a person to say yes.

There is a second reason. In AI systems many trade-offs are numbers: accuracy against cost, speed against quality. "We chose the cheaper model" is useless a month later. "We accepted 91% instead of 94% to halve the cost per question" (made-up numbers) can be checked and argued with.

It is a short file that compares the options against the team's real situation, picks retrieval, and admits what retrieval costs. The new engineer's question became ADR-0007, and this time the answer was tested and written down.

First, the two options in plain words. Fine-tuning trains an existing model further on your examples, so its habits and knowledge shift toward your data. Retrieval means your code first searches your own documents for the parts that match the question, then pastes them into the prompt.

The 2020 paper on retrieval-augmented generation (RAG) states the problem well. Models store facts in their parameters, the huge set of numbers learned in training, but using that knowledge precisely, and updating it, is hard. Retrieval adds an outside memory the system can search; here, that is the policy handbook.

Pause and predict. The fine-tuned trial scored 84% and the retrieval trial scored 93%. Before you read the file, guess which single fact about this team settles the choice, even if the scores were equal.

Here is ADR-0007 as the team wrote it, in the Markdown file the scripts later in this guide read. Every number in it is made up.

# 7. Use retrieval, not fine-tuning, for policy answers

## Status

Accepted, 2026-04-18

## Context

The policy handbook is about 120 pages and changes about twice a month.
Sellers must get today's rule, with a link to the page it came from.
Nobody on the team has trained a model before.

## Options

1. Fine-tune a model on the handbook: 84% on the eval set, about Rs 40,000 per training run, stale after every policy change.
2. Retrieval: search the handbook, put the top matching sections in the prompt: 93% on the eval set.
3. Paste the whole handbook into every prompt: 90%, about four times the cost per question.

## Decision

We will use retrieval (option 2) for all policy questions.

## Consequences

Good: a policy change goes live the same day by re-indexing one document.
Good: answers can show which handbook page they used.
Bad: the search step adds about half a second per answer.
Bad: if search misses the right page, the answer is wrong, so search needs its own tests.

## Confirmation

Eval set: policy-questions, version 3, 300 questions.
Eval threshold: 0.90
Result: evals/results/2026-04-17.json (279 of 300 passed)
Revisit when: policies stop changing often, or sellers need a writing style that prompting cannot teach.

What just happened? The settling fact sits in the context: the handbook changes about twice a month. A fine-tuned model goes stale after every policy change, so retrieval wins even at equal scores. The context, not the score, decided it.

Notice the honest Bad lines. Notice too the Revisit when line, which tells a future reader exactly what would make this decision wrong. The new engineer's question now has a one-minute answer: read ADR-0007.

Beginners can stop here. You now know what an ADR is, which sections it has, and what a real AI decision looks like on paper. The rest of the guide covers approvals, automatic checks, templates and superseding.

It records which AI actions need a person to say yes before they happen, why, and what that costs in speed. For the team, this became ADR-0011, written after the assistant was given the power to change prices.

In June, the team added a tool to the assistant: a function it may ask the app to run. This one updates prices. A seller can type "Put all my soaps on 10% off for Dasara" and the assistant prepares the change.

During a one-week trial (made-up numbers), the assistant drafted 2,000 price changes. Fourteen were wrong. One set a Rs 1,500 sandalwood gift box to Rs 150.

Security guidance points the same way. OWASP, an open security community, publishes the OWASP Top 10 for LLM Applications. It lists security risks for apps built on an LLM, a large language model. Its 2025 edition names Excessive Agency as risk LLM06. Its mitigations include human approval of high-impact actions before they are taken. They also say to enforce authorisation in downstream systems, the systems that carry out the action such as the price service, rather than letting the model decide.

Here is the short version of the record:

• Title: 11. Seller must approve every price change the assistant drafts.
• Status: Accepted, 2026-07-02.
• Context: The assistant can now draft price changes. In the trial, 14 of 2,000 drafts were wrong, and one cut a price by 90%.
• Options: let the assistant act alone with limits in code; ask the seller to approve every change; ask only for risky changes.
• Decision: The assistant only drafts. The seller sees old and new prices and taps Approve. Code rejects any price under Rs 10 or any cut over 50%, whatever the model asks.
• Consequences: no price changes without a seller's tap. Sellers take one extra step, and drafts nobody approves expire after 24 hours.

The diagram shows the flow ADR-0011 describes. Follow the path from the seller's message to the price list, and notice that the model never touches prices directly.

The ADR does not explain how to build the approval screen; that belongs in the code. It explains why the screen exists, so nobody removes it later just to make the feature feel faster.

You write the eval set, its version, the threshold and the results file inside the ADR. Then an automatic check compares every new change against that threshold. The ADR holds the promise; the eval results prove it is still kept.

Go back to the March incident. Later, the team re-ran the eval set on that cheaper model, and it passed 261 of 300 questions (made up), which is 0.87. ADR-0007 needs 0.90, so the change would have stopped before reaching any seller.

The team stores each eval run as a small JSON file (a simple text format for data) with two counts, passed and total. The script below runs on every pull request, a request to merge a change that teammates review. Automatic checks like this are part of CI (continuous integration), the practice of building and testing every change automatically. In CI, the job runs the eval set itself and writes this file. A results file typed by hand, or committed with the change, proves nothing.

import json
import math
import re
import sys
from decimal import Decimal


def threshold_from_adr(adr_path):
    with open(adr_path, encoding="utf-8") as f:
        text = f.read()
    match = re.search(r"Eval threshold: ([0-9]+[.][0-9]+)", text)
    if match is None:
        sys.exit(f"FAIL: no 'Eval threshold: 0.NN' line in {adr_path}")
    return Decimal(match.group(1))


def check_release(adr_path, results_path):
    threshold = threshold_from_adr(adr_path)
    with open(results_path, encoding="utf-8") as f:
        results = json.load(f)
    passed, total = results["passed"], results["total"]
    if not total:
        sys.exit(f"FAIL: {results_path} has no questions")
    needed = math.ceil(threshold * total)
    short_by = max(0, needed - passed)
    if short_by:
        sys.exit(f"FAIL: {passed} of {total} passed, {adr_path} needs {needed}")
    print(f"OK: {passed} of {total} passed, {adr_path} needs {needed}")


check_release(sys.argv[1], sys.argv[2])

The function threshold_from_adr finds the line "Eval threshold: 0.90" in the ADR. In the pattern, [0-9]+ means one or more digits and [.] means a real dot. If the line is missing, or the threshold is not written as a decimal like 0.90, the script fails loudly instead of passing by accident.

Next, check_release turns the threshold into a count of questions. For 300 questions at 0.90, that is 270. The variable short_by is how many passes are missing. Any value other than zero ends the script with exit code 1, and a non-zero exit code tells CI the check failed. An empty results file fails too.

Why the script counts questions instead of comparing decimals

The obvious version divides passed by total and compares decimals. That comparison is correct, but its message misleads. Printing 269 of 300 to two decimals shows 0.90, so a message like "0.90 is below 0.90" confuses everyone. Printing counts avoids that.

Counting brings its own trap, because the threshold must become a count first. Ordinary floating-point maths, the way code usually stores decimals, is not exact. In Python 3.13, 0.81 times 300 gives 243.00000000000003, so rounding that up with math.ceil would demand 244 passes instead of 243. Reading the threshold as a Decimal keeps 0.81 exact, so the count is 243.

How to check the gate works

Save the ADR at docs/decisions/0007-retrieval-over-fine-tuning.md and the script as check_eval.py. For this local test only, put {"passed": 279, "total": 300} in evals/results/latest.json. Then run the script with the ADR path and the results path.

• Command: python3 check_eval.py docs/decisions/0007-retrieval-over-fine-tuning.md evals/results/latest.json
• With 279 passes, expect "OK: 279 of 300 passed, docs/decisions/0007-retrieval-over-fine-tuning.md needs 270" and exit code 0.
• Change passed to 270 and it still prints OK, because 270 of 300 is exactly 0.90.
• Change passed to 269 and it prints a line starting "FAIL: 269 of 300 passed", with exit code 1.
• Run echo $? straight after the script on Linux or macOS to see the exit code.
• On Windows, type py instead of python3, then run echo $LASTEXITCODE in PowerShell or echo %ERRORLEVEL% in cmd.
• FAIL lines go to standard error, so they show in the terminal and CI logs but not in a file that captures only normal output.

Because the threshold lives in the ADR, lowering it is not a quiet config change, as long as ADR edits go through pull request review. It shows up in a pull request as an edit to an accepted decision, which the team's rule forbids. To accept a lower score, someone proposes a new ADR and points the check at it.

Start with Nygard's five sections, then borrow from MADR when the team wants options and a confirmation step. The three well-known templates agree on the core and differ on details that matter for tooling and for editing old records.

Front matter is a block of key and value lines at the very top of a Markdown file, between two lines of three dashes. MADR keeps status, date and decision-makers there, so it has no Status heading.

The templates disagree about old records. Nygard keeps an old decision and marks it superseded, because it was the decision, even if it no longer is. The UK framework asks teams to regularly review and update the ADR as its context or consequences change. It does not mention superseding at all.

The team settled it with one rule. Status lines and links may be updated; the decision and its reasons are never rewritten. A changed decision gets a new ADR.

The UK framework is meant for the whole UK public sector, but it is mandated only for cross-government and cross-public-sector decisions. As of 19 October 2026, MADR 4.0.0 is still the latest release of the madr package on npm, the JavaScript package registry.

You write a new ADR that explains what changed and why, then edit only the old ADR's status line to say "Superseded by" the new one. You never rewrite the old decision itself.

ADR-0011 shows how this works. Three months after launch (made-up numbers), sellers approved small drafts unchanged 98 times in 100. About 1 in 5 sellers stopped using price changes because of the extra tap. Meanwhile the hard limits in code had blocked every bad draft the team found.

So in October an engineer proposed ADR-0016. Changes of 15% or less, on 10 listings or fewer, apply at once and can be undone for 24 hours. Bigger changes still need the seller's tap, and the hard limits stay.

Once ADR-0016 was accepted on 9 October, ADR-0011's status line became "Superseded by ADR-0016". Nothing else in it changed. Anyone can now read both files and see what the team believed in July, and what evidence changed its mind.

The diagram shows the statuses an ADR can move through. Look at where ADR-0011 ends up, and notice that no arrow leads to deleted.

Rejected ADRs are worth keeping too. Say the team writes "ADR-0009: Let the assistant reply to buyer complaints on its own" and rejects it. The next person with that idea can read why first, instead of repeating the whole argument.

Keep ADRs as Markdown files in the same repository as the code they describe, in one folder, numbered in order. MADR suggests a docs/decisions folder and names like 0007-retrieval-over-fine-tuning.md: a running number, then the title in lower case with dashes.

Keeping them in the repository has three benefits. They are reviewed in the same pull request as the code they explain. They carry the same history as the code. And an engineer reading the retrieval code is one click away from the reason it exists.

Files in a folder drift unless something checks them. The team's CI runs a second small script. It checks that each ADR has the four core headings, each on a line of its own, and that every "Superseded by" line points to an ADR that exists.

import re
import sys
from pathlib import Path

REQUIRED = ["## Status", "## Context", "## Decision", "## Consequences"]


def check_decisions(folder):
    files = sorted(Path(folder).glob("[0-9][0-9][0-9][0-9]-*.md"))
    numbers = {f.name[:4] for f in files}
    problems = []
    for f in files:
        text = f.read_text(encoding="utf-8")
        for heading in REQUIRED:
            if not re.search(rf"^{re.escape(heading)}\s*$", text, re.MULTILINE):
                problems.append(f"{f.name}: missing {heading}")
        for new_number in re.findall(r"Superseded by ADR-([0-9]{4})", text):
            if new_number not in numbers:
                problems.append(f"{f.name}: points to missing ADR-{new_number}")
    return files, problems


files, problems = check_decisions(sys.argv[1])
for p in problems:
    print(p)
print(f"{len(files)} ADRs checked, {len(problems)} problems")
sys.exit(1 if problems else 0)

A glob pattern is a simple filename filter, where [0-9] means any one digit and * means anything. This one picks only files that start with four digits, so a stray notes.md is ignored. The numbers set holds every ADR number that exists, such as 0011 and 0016. The heading test uses re.MULTILINE, so ^ and $ match the start and end of each line. A line like ## Statuses therefore does not count as ## Status.

The script expects Nygard-style headings, exactly as written, such as ## Status. MADR uses different headings, such as ## Decision Outcome. It keeps status in front matter, in lower case, such as "superseded by ADR-0123". A MADR team would check its own headings and the front matter instead.

To check it, save ADR-0007, ADR-0011 and ADR-0016 with those four headings, then run python3 check_decisions.py docs/decisions (py on Windows). Expect "3 ADRs checked, 0 problems" and exit code 0. Type ADR-0061 instead of ADR-0016 in ADR-0011's status line, and the output gains a line ending "points to missing ADR-0061", with exit code 1.

Here is how the team writes a new ADR, from idea to merge:

1. 1

   Take the next number

   Copy the template to the next free number, for example 0017. Numbers are never reused, even for rejected ADRs.

2. 2

   Write the context first

   Describe the situation with real numbers before naming any option. If you cannot explain the problem, you are not ready to decide.

3. 3

   List at least two options

   Include the ones you did not pick, with their scores or costs. Doing nothing is often a fair option.

4. 4

   Attach the evidence

   Run the eval set, save the results file, and write the threshold you are proposing in the ADR.

5. 5

   Open a pull request with status Proposed

   Put the ADR in the same pull request as the code change, or before it, so reviewers judge both together.

6. 6

   Accept and link

   When the team agrees, set the status to Accepted with a date, and mention the ADR number in the code and prompt files it governs.

The team did one more useful thing. It back-filled ADRs 0001 to 0006 for the biggest decisions made before April, writing down only what it could still confirm. A small, honest log beats a large, guessed one.

Treat every big choice in your design as a small ADR: name the decision, the options, the evidence and the cost. Interviewers are checking that you can reason about trade-offs, not that you know a template.

1. 1

   Name the decision

   Say it in one line, such as "How does the assistant learn the marketplace's rules?"

2. 2

   Give the context

   Mention the facts that matter: how often the data changes, who uses the answers, and what a wrong answer costs.

3. 3

   Compare two or three options

   For example retrieval, fine-tuning, or putting everything in the prompt, each with one strength and one weakness.

4. 4

   Decide and admit the downside

   Pick one and say what gets worse, such as a search step that needs its own tests.

5. 5

   Say how you would check it

   Name the eval set, the threshold, and the condition that would make you revisit the decision.