AI SLA: How to Turn AI Trade-offs into Promises You Can Keep

A bare promise of 99% accuracy goes wrong because nobody can check it. It names no unit, no sample, no period and no remedy, so the first dispute has no answer.

Picture a common setup. A document extraction service reads invoices and shipping bills for freight companies and pulls out key fields: invoice number, date, total amount and GSTIN. A GSTIN is the 15-character registration number of an Indian business under GST (goods and services tax). The setup and its numbers are made up, but the problem is real.

One freight client sends the service about 40,000 documents a month. The first contract had one quality line, written in a hurry by sales: the service guarantees 99% accuracy. For two months, nobody looked closely.

Then the model provider released a new version of its model. The model provider is the outside company that runs the AI model the service calls for each page. For three days, wrong GSTINs went up sharply, and the client's accounts team found about 300 invoices with the wrong number.

The client said the service had promised 99%. The engineering manager opened the contract and could not answer. Nobody could say whether 99% meant documents, fields or characters, who measured it, over what period, or what a miss would cost.

This guide rebuilds that promise step by step. By the end, the team has numbers it can measure, targets it can keep, a plan for when the model fails, and a monthly report the client trusts.

An SLI is what you measure, and an SLO is the target you set for yourself. An SLA is the promise you sign with a client, including what happens if you miss it. The three build on each other, in that order.

Think of a tiffin delivery service. The SLI is how late each tiffin arrives. The SLO is the delivery team's own goal: 98 out of 100 tiffins before 1 pm. The SLA is the line in your monthly plan: if more than 5 tiffins are late in a month, you get two days free.

Where the picture breaks: a late tiffin is obvious to everyone. A wrong AI answer looks exactly like a right one, so somebody has to check outputs before a failure can even be counted.

SLI. A service level indicator is a number you measure about your service. Example: the share of documents that were ready within 30 minutes of upload yesterday.

SLO. A service level objective is the target you set for an SLI, over a fixed period. Example: 99% of documents ready within 30 minutes, measured over each calendar month. The SLO is for your own team, and it is usually stricter than the client promise.

SLA. A service level agreement is the contract with a client. It contains a few targets, says exactly how they are measured, and names the remedy for a miss. A common remedy is a service credit: money taken off the next bill.

Error budget. The error budget is the amount of failure your SLO allows. If the target is 99% and the client sends 40,000 documents, 1% of them, which is 400 documents, may be late before you miss the target.

Golden set. A golden set is a fixed pile of real documents where people have already written down the correct answer for every field. Example: 500 of the client's past invoices with checked values. You run the system on it to get a quality score you can compare over time.

Fallback. A fallback is the backup path your system takes when the normal path fails. Example: if the model provider is down, the document waits in a queue (a waiting line of jobs) and the service tries a second provider.

Google's SRE book (site reliability engineering, the practice of running services reliably) uses the same three words. It adds a useful test: if missing the target has no explicit consequence, you are almost certainly looking at an SLO, not an SLA.

The diagram shows how the words connect for one number. Read it from left to right: a measurement feeds a team target, and the team target protects a client promise.

An AI SLA must handle two extra problems: answers that are right only most of the time, and a model that can change under you. A normal service is mostly up or down. An AI service can be up, fast, and still quietly wrong.

The team found four differences when it rewrote the contract.

• Quality is a rate. On a messy scan, the model may read a GSTIN correctly most times and wrongly some times. You can only promise a rate, measured on many documents.
• The same input can give different outputs. A 2024 study ran five models on eight tasks, 10 times each, with settings meant to make them repeatable. Accuracy still varied by up to 15% between runs.
• The model can change without you changing any code. A 2023 study compared March and June 2023 versions of two hosted models sold under unchanged names. One model's accuracy at telling prime from composite numbers fell from 84% to 51%.
• You depend on someone else's service. If the model provider is slow or down, your service is slow or down, unless you planned a fallback.

The second point means one bad document does not prove a broken system, and one good document does not prove a working one. The third point is exactly what hit the GSTIN field in the opening.

The model update problem needs its own process. A model API is the web address your code calls to use a model. As of 19 October 2026, large hosted model APIs commonly offer dated model versions you can pin, and publish retirement dates for old ones. The team had not pinned a version, so the new one arrived with no warning.

Pinning only delays change, because pinned versions are retired too. What the team can control is when and how a new version reaches the client.

The diagram shows the team's gate for any model change. A new version must pass two checks before it touches all of the client's documents. The second check is a canary release: a small share of real traffic goes to the new version first, so a problem hurts few documents.

If a check fails, the team may change the prompt, which is the instruction text the service sends to the model with each page. With this gate, the bad GSTIN reads would have shown up in the golden set or the canary, not in the client's accounts.

You measure AI quality by having people check a random sample of real outputs against the true values, and reporting the share that was correct. The rule for what counts as correct must be written down before you measure.

The old promise failed because it named no unit. The team replaced it with something precise. The unit is a field, not a whole document. Only critical fields count: GSTIN, invoice number, invoice date, total amount and shipping bill number.

A field is correct only if it matches the true value exactly, after removing extra spaces and writing dates one way. Here is how the team builds its quality SLI, step by step.

1. 1

   List the fields that matter

   Ask the client which mistakes cost money. A wrong GSTIN means hand corrections in the accounts. A wrong consignee address line does not, so it stays out of the SLA.

2. 2

   Write the rule for correct

   Exact match after removing spaces, and the date must be the same calendar day whatever its format. Put this rule in the contract.

3. 3

   Pick a fair sample

   Each week, choose 2% of documents at random. Two reviewers check each critical field, and a third settles any disagreement.

4. 4

   Decide what is excluded

   Pages the service flags as unreadable go to a person and are left out of the quality number. Their count is still reported, so nobody can hide failures in this group.

5. 5

   Fix the period

   Report the score for each calendar month, so a single bad day does not decide the whole contract.

The Python below turns audited documents into the quality SLI. Each audited document holds what the service extracted, what the reviewers agreed is true, and an unreadable flag. Assume the values were already cleaned: spaces removed and dates written one way.

from dataclasses import dataclass

CRITICAL = ["gstin", "invoice_no", "invoice_date", "total", "bill_no"]

@dataclass
class AuditedDoc:
    extracted: dict      # what the service returned
    true_values: dict    # what the reviewers agreed is correct
    unreadable: bool = False

def critical_field_accuracy(audited_docs):
    checked = 0
    correct = 0
    for doc in audited_docs:          # random sample checked by people
        if doc.unreadable:            # excluded, but counted in the report
            continue
        for field in CRITICAL:
            if field not in doc.true_values:  # not on this document
                continue
            checked += 1
            if doc.extracted.get(field) == doc.true_values[field]:
                correct += 1
    if checked == 0:
        return None                   # no audit data: report it, do not guess
    return correct, checked

sample = [
    AuditedDoc(  # an invoice: no shipping bill number on it
        extracted={"gstin": "29ABCDE1234F1Z5", "invoice_no": "INV-881",
                   "invoice_date": "2026-09-14", "total": "41300.00"},
        true_values={"gstin": "29ABCDE1234F1Z5", "invoice_no": "INV-881",
                     "invoice_date": "2026-09-14", "total": "41300.00"}),
    AuditedDoc(  # one digit of the GSTIN misread, bill number missed
        extracted={"gstin": "29ABCDE1234F1Z8", "invoice_no": "INV-882",
                   "invoice_date": "2026-09-15", "total": "9800.00"},
        true_values={"gstin": "29ABCDE1234F1Z5", "invoice_no": "INV-882",
                     "invoice_date": "2026-09-15", "total": "9800.00",
                     "bill_no": "7731906"}),
    AuditedDoc(extracted={}, true_values={}, unreadable=True),
]

correct, checked = critical_field_accuracy(sample)
print(f"{correct} of {checked} critical fields correct: {correct / checked:.1%}")

CRITICAL is the list of fields that count. The outer loop skips unreadable documents. The inner loop skips critical fields that are not on that document, since an invoice has no shipping bill number. It then compares each remaining field with the reviewers' value.

The .get() call returns None when the service extracted nothing for a field. So a missed field counts as wrong instead of crashing the script. If nothing was checked, the function returns None, because a month with no audit is missing data, not 100%.

To check it, save the file and run python3 quality_sli.py (we ran it on Python 3.13). It should print one line: 7 of 9 critical fields correct: 77.8%. The misread GSTIN and the missing bill number are the two wrong fields.

Now the real month, with made-up numbers. The team audits about 800 documents (2% of 40,000). After removing unreadable pages, they hold 3,600 critical fields, and 3,537 are correct, so the SLI is 98.25%.

The measured score is below the team's own 98.5% SLO, so the team acts. The SLA promises 97.5%, so the client is still within the contract.

Set availability as the share of valid requests your service handles successfully. Set latency as the time by which most requests finish, such as 95 out of 100. Then check that your suppliers can support those numbers before you promise them.

Latency means waiting time. Here it is the time from upload to result ready. The team uses percentiles rather than averages. The p95 is the time within which 95 out of 100 documents finish. An average can look fine while a few documents wait an hour.

Availability means the share of valid requests that succeed. Valid matters. A document in an unsupported file type, or one sent by a program that is not logged in, should not count against the service. The contract must say what makes a request valid.

This work is not a live chat. The client uploads documents, and results come back a few minutes later. So the team sets two separate targets, with made-up numbers, both measured at its own front door (the point where uploads enter its system) per calendar month.

• Upload availability: the share of valid uploads that the upload API (the web address the client's software sends documents to) accepts. The team aims for 99.95% and the contract promises 99.9%.
• Time to result: the team aims for 95% of documents within 5 minutes and 99% within 30 minutes. The contract promises 97% within 30 minutes. This target depends on the model provider too.

Splitting them matters. Say the team's own parts work 99.9% of the time and the model provider works 99.5% of the time. If every request needed both at the same moment, the service could be up only about 99.4% of the time.

That is 0.999 times 0.995, which is 0.994, assuming the two fail independently. A chain of required parts is never more available than its weakest part.

Uploads go into the queue first, so a model outage does not block an upload. It only delays the result. The team can therefore promise 99.9% upload availability and absorb short model outages inside the time-to-result target.

An error budget is the number of failures your SLO allows in a period, and you spend it like money. While budget is left you can ship changes; when it runs out, you pause risky changes and fix reliability. For AI, model and prompt changes are the risky ones.

Back to the tiffin service. If the goal is 98 out of 100 on time, then 2 late tiffins in 100 is the budget. Spend it on a new route trial if you like, but once it is gone, no more experiments this month.

Google's SRE workbook gives an example error budget policy. When a service is over budget, it freezes all changes except top-priority fixes and security fixes until the service is back within its SLO.

Here the time-to-result SLO is 99% within 30 minutes. With 40,000 documents a month, 400 may be late. The team writes down in advance what happens at each level, so nobody argues about it during a bad week.

The Python below works out how much of that budget is left and prints the rule that applies. The numbers are made up: 310 documents have been late so far this month.

def budget_left(total_docs, bad_docs, target=0.99):
    allowed = total_docs * (1 - target)    # failures the SLO allows
    return (allowed - bad_docs) / allowed  # 1.0 = untouched, 0 or less = spent

left = budget_left(total_docs=40_000, bad_docs=310)
if left <= 0:
    print("Budget spent: freeze model and prompt changes")
elif left < 0.25:
    print(f"{left:.1%} of budget left: only small, tested changes")
else:
    print(f"{left:.1%} of budget left: normal releases")

The function first works out how many failures the target allows: 1% of 40,000 is 400. It then returns the share of that allowance still unused: (400 minus 310) divided by 400, which is 0.225.

Run it and it prints 22.5% of budget left: only small, tested changes. So this month the team ships only small changes that passed the golden set, and a new prompt for shipping bills waits.

The team keeps a second budget for quality. With a 98.5% field accuracy SLO, 1.5% of audited critical fields may be wrong: 54 of 3,600. This month 63 were wrong, so that budget is spent and model and prompt changes are frozen anyway.

When two budgets disagree, the stricter rule wins. The diagram shows the rule the team follows for each budget; the two lower branches turn a number into an action.

When the model provider is down, your system should switch to a planned fallback and never fail silently. Common fallbacks are retrying, using a second provider, or holding work in the queue. Each fallback must be tested before the outage, not invented during it.

NIST, the US government's standards and technology agency, publishes a risk profile for generative AI (AI that produces text or images). It asks teams to identify fallbacks for third-party AI, and to test them. It notes a fallback may include manual processing.

Here is the order the service tries things, from cheapest to most expensive.

A few details make this work in practice.

• Retry with growing, slightly random waits, such as about 2, then 8, then 30 seconds. Google's SRE book says to always use randomised exponential backoff and to limit retries per request.
• Do not retry errors that can never succeed, such as a page the provider rejects as malformed. Send those straight to a person.
• Score the backup provider on the golden set every month. If its GSTIN accuracy is lower, tag its results as backup path so the client's team knows to check them.
• Check that the backup is allowed. If the contract says the client's data must stay in India, a backup in another country is not a fallback, it is a breach.
• Let documents that are not urgent wait in the queue. The 30-minute target gives room for a short outage.

An AI SLA should promise things you can measure and control: availability, time to result and quality on an agreed sample. It should leave out promises about single outputs, perfection, or a specific model that you cannot control.

NIST's cloud service metrics guide makes the same point for cloud services. Agreements are built from measurable objectives, and if they are not met, a remedy decided in advance applies. Here is how the team split the new contract, with made-up numbers.

Notice that the contract never names the model. The service promises results, not a supplier, which keeps it free to switch providers when the gate says the new one is as good.

Also notice that every promise sits below an internal target: 97.5% promised against 98.5% aimed for on quality. That gap is room for a bad month without a broken contract.

Check your SLOs weekly inside the team, report to clients monthly, and review the targets themselves every quarter, plus after every model change or incident. NIST's generative AI profile asks teams to plan ongoing monitoring and periodic review of their AI risk management, and to decide how often. It does not set a schedule.

The team's rhythm looks like this. Every week, it spends 30 minutes on the SLIs and both error budgets. Every month, the client gets a report. Every quarter, both sides ask whether the targets are still right, for example whether a new field matters or 97.5% is too easy.

Some events trigger an extra review straight away: any model change, any missed SLO, and any complaint like the GSTIN one. Waiting for the next quarter after a bad incident only lets the same mistake happen again.

The monthly report is one or two pages and always has the same six parts.

• Each SLA number next to its target, with a plain met or missed.
• How much of each error budget was used.
• How many documents were excluded as unreadable, and how many took a fallback path.
• Every incident: what happened, how many documents were affected, the cause and the fix.
• Model or prompt changes made this month, with golden set scores before and after.
• Planned changes for next month, so nothing reaches the client as a surprise.

The report builds trust because it shows bad news too. After the GSTIN problem, the client did not need the service to be perfect. It needed to hear about problems from the team first, with numbers, before its accounts team found them.

Beginners can stop here. You can now turn a vague accuracy promise into an SLI, an SLO and an SLA with a remedy. The next two sections are for engineers who have to defend those numbers.

A quality audit must be big enough that the lower edge of its uncertainty band stays above your promise. A sample gives an estimate, not the true rate, so report the band with the number.

A 95% confidence interval is a band built so that, over many repeated audits, about 95 in 100 such bands contain the true rate. The Wilson score interval is a standard way to compute one for a share. The NIST statistics handbook gives its formula.

from math import sqrt

def wilson_interval(correct, checked, z=1.96):
    """95% Wilson score interval for a share, as (low, high)."""
    p = correct / checked
    centre = p + z * z / (2 * checked)
    spread = z * sqrt(p * (1 - p) / checked + z * z / (4 * checked * checked))
    bottom = 1 + z * z / checked
    return (centre - spread) / bottom, (centre + spread) / bottom

for correct, checked in [(3537, 3600), (177, 180)]:
    low, high = wilson_interval(correct, checked)
    print(f"{correct}/{checked} = {correct / checked:.2%}, "
          f"95% interval {low:.2%} to {high:.2%}")

Run it with python3 audit_interval.py and it prints two lines. The first is 3537/3600 = 98.25%, 95% interval 97.77% to 98.63%. The second is 177/180 = 98.33%, 95% interval 95.22% to 99.43%.

What just happened? The monthly audit of 3,600 fields puts the true rate between 97.77% and 98.63%. The 97.5% promise is met, though not by much: the lower edge clears it by about a quarter of a point. The 98.5% SLO sits inside the band, so the team cannot be sure it missed, and it acts anyway because acting early is cheap.

The second line is a small client sending 2,000 documents a month. A 2% sample is 40 documents, about 180 fields. Even with 177 correct, the band runs from 95.22% to 99.43%, so the audit cannot say whether a 97.5% promise was kept.

For small clients, audit a fixed minimum number of fields each month rather than a percentage, and write that minimum into the contract.

Errors bunch up on bad scans

The Wilson interval treats every field as an independent coin toss. Real errors bunch up: one blurry scan can get four fields wrong at once. That makes the true uncertainty wider than the formula says.

We checked this with a made-up simulation in which 3% of documents were bad scans. Across four runs of 300 simulated audits of 800 documents, the field-level band contained the true rate only 86% to 89% of the time, not 95%.

A bootstrap did better. It rebuilds the audit many times by drawing whole documents at random, with repeats allowed. Its bands were about a fifth wider and contained the true rate 91% to 95% of the time. These figures come from our script, not a published study.

The practical rule: resample whole documents when you compute the band. Keep the gap between SLO and SLA wider than half that band, so a month that meets the SLO also clears the SLA.

A burn-rate alert fires when you are spending error budget much faster than your target allows, so you hear about a bad hour before it becomes a bad month. Burn rate is how fast you use the budget compared with the SLO.

At a burn rate of 1, a steady failure rate uses exactly the whole budget by the end of the period. At a burn rate of 10, a 30-day budget is gone in 3 days.

For the 99% time-to-result SLO, a burn rate of 14.4 means 14.4% of documents are late. Kept up for one hour, that spends 2% of a 30-day budget. One hour is 1/720 of 30 days, and 14.4 divided by 720 is 0.02.

Google's SRE workbook suggests that as a starting point for paging someone. Its example uses a 99.9% SLO, but burn rate is relative to the SLO, so the factors carry over. The alert checks the last hour and the last 5 minutes together, so it also stops soon after the problem ends. It suggests a ticket, not a page, for a burn rate of 1 over 3 days.

Burn-rate alerts suit time to result and availability, which you measure on every document. The quality SLI comes from a weekly audit, which is too slow for paging. Between audits, watch cheap automatic signals, such as the share of GSTINs that fail their check digit.

Explain it as a chain: what you measure, the target you set yourself, the promise you make, and what happens when things fail. Interviewers want to see that you can turn a wish for accurate and fast into numbers someone can check.

1. 1

   Pin down the unit of quality

   Say what counts as correct and for which fields, like the five critical fields with exact match.

2. 2

   Name the SLIs

   Pick a few: availability of the front door, time to result at p95, and quality on a random audited sample.

3. 3

   Set SLOs stricter than the SLA

   Explain the gap, the error budget each SLO creates, and what the team does when a budget is spent.

4. 4

   Size the audit

   Say how many fields you check each month and how wide the uncertainty band is, so the quality number can be enforced.

5. 5

   Handle model change

   Describe the golden set, a canary release and advance notice to clients before any model switch.

6. 6

   Plan for failure

   Walk through the fallback path when the model provider is down, and say why a late checked answer beats a fast guess.

7. 7

   Close with review and reporting

   Burn-rate alerts, weekly checks, monthly client reports and quarterly target reviews.