Human in the Loop AI: Designing Escalation and Approvals

A wrong decision goes out at machine speed, and nobody notices until a customer complains or the money is gone. Human in the loop AI is the usual answer: a person must approve the risky actions before they happen.

Picture a common setup. A motor insurance claims team in India pays for car repairs. A customer reports that an auto-rickshaw hit the back of her hatchback. She uploads six photos of a dented rear bumper and a garage estimate of Rs 38,000. The setup is made up, but the problem is real.

A claims officer picks up the claim and checks that the policy was active on the day of the accident. Next come the photos. The officer checks that the damage is really at the back and looks fresh, because rust inside the dent would mean old damage.

Then each line of the estimate is checked against the team's parts price list. The bumper is Rs 21,000, paint Rs 9,000 and labour Rs 5,000. A Rs 3,000 polishing charge is not covered, so it comes off. So does the Rs 2,000 deductible, the part of each claim the customer pays. The officer approves Rs 33,000 after about 12 minutes.

The trouble is volume. The team gets 300 new claims a day, and four officers can clear about 160, so customers now wait nine days for an answer. An engineer on the team builds an AI assessor. It runs on a model, the trained program that turns photos and text into an answer.

The assessor reads the photos, lists the damaged parts, looks up prices and proposes a payout in seconds. That speed creates the real question. If it pays Rs 33,000 by itself and it is wrong, who notices? This guide answers that: which claims still need an officer, how they get there, and how to make the officer's yes mean something.

Six words carry this guide. Each gets a plain meaning and an example from the claims team.

Human in the loop (HITL). A person must approve the AI's proposal before it becomes a real action. Example: the AI proposes Rs 33,000, and no money moves until an officer clicks Approve.

Escalation. The system takes a case off the automatic path and hands it to a person, along with the reason. Example: escalated because a photo was taken three months before the accident date.

Approval gate. A point in your code where an action waits for a person's yes. Example: the pay_claim function refuses to pay an escalated claim unless the audit log holds a reviewer's approval for that claim.

Because the gate sits in the payment code itself, a prompt (the instruction text sent to the model) that fools the AI cannot get past it. That holds only if every payment goes through that one function, and only the reviewer screen, after the officer signs in, can write an approval.

Confidence score. A number, often from 0 to 1, that the AI system attaches to its answer. Higher should mean more likely to be right, but you have to check that. A threshold is the cut-off you compare it with, such as below 0.90 goes to a person.

Reviewer queue. The list of cases waiting for a person, sorted so the most urgent come first. Example: 40 claims waiting, with a customer whose car is stuck at a garage at the top.

Audit trail. A permanent record of what was decided, by whom or by what, when, and on which evidence. Example: claim 7781, AI proposed Rs 33,000 at 0.94, officer 12 approved Rs 33,000 at 4:12 pm.

Human in the loop AI means a person approves each action before it happens. Human on the loop means the AI acts by itself while a person watches and can step in or reverse it. Human out of the loop means nobody checks each case.

Think of a kirana shop, a small neighbourhood grocery. The helper sells a packet of biscuits alone and the owner never looks: out of the loop. The helper bills a bigger order while the owner watches from the counter: on the loop. A customer wants Rs 2,000 on credit, so the helper must ask the owner first: in the loop.

Where the picture breaks: the owner can see the whole shop, but a reviewer sees only what the screen shows. And an AI makes hundreds of decisions a day, not a few dozen.

The claims team uses all three levels, for different jobs. The table shows where each one fits.

On the loop is not in the loop with less effort. The money has already gone when the officer looks. So on the loop fits only actions that are small and that you can still put right, for example by asking the garage to return an overpayment.

Ask four questions about the worst likely mistake: how much harm, how much money, can it be undone, and how often the AI is right on cases like this. If any answer is bad, put a person in the loop.

• Risk of harm: who gets hurt if the AI is wrong, and how badly? Wrongly rejecting an honest claim hurts a real person who paid for cover.
• Money: how much moves in one action? Rs 8,000 for a scratch is very different from Rs 4 lakh (400,000) for a car that cannot be repaired.
• Reversibility: can you undo it cheaply? A note in a file is easy to fix. Money paid out, or a rejection letter the customer has already read, is not.
• Accuracy on similar cases: measure it on past claims that officers checked, not on the AI's own score. The engineer section shows how.

Ask about the worst mistake, not the average one. Most bumper claims are boring. The rule exists for the one claim whose photos come from an old accident.

One rule surprises many beginners: the team never lets the AI reject a claim alone. A wrong payout costs the insurer money. A wrong rejection costs an honest customer their cover and their trust, so the two directions of mistake get different rules.

For customers in the EU, the law points the same way. GDPR (the EU's data protection law) gives people a right, in Article 22, not to face a significant decision based solely on automated processing. Where its contract or explicit-consent exceptions apply, people keep at least the right to human intervention and to contest the decision.

OWASP, a non-profit security community, gives the same advice for any AI that can take actions. It keeps a 2025 list of top risks for LLM applications (apps built on large language models). One risk is Excessive Agency: an AI given more functions, permissions or autonomy than it needs. One of its fixes is to have a person approve high-impact actions before they are taken.

Here is how the team routes each claim after the AI has assessed it. Start at the top and follow the answers. Notice that paying without review is the last path reached.

Run the bumper claim through it. Rs 33,000 is under Rs 1.5 lakh, with no fraud signs, a score of 0.94 and an old policy. But it is over Rs 15,000, so it goes to an officer. A Rs 9,000 scratch scored 0.96 would be paid at once, and might be one of the 5 in 100 checked later.

An escalation trigger is a written rule that sends a case to a person. Good triggers come from three places: your risk limits, signs that the AI may be wrong, and signs that the case is unusual.

The team's triggers, all made up for this example, are:

• Limits: the proposed payout is over Rs 15,000, or the AI suggests rejecting the claim.
• Low confidence: the AI's score is below 0.90.
• Fraud signs: a photo's saved date is before the accident, or the same photo appeared in an earlier claim.
• Mismatch: the customer says "hit from behind" but the photos show front damage.
• Unusual input: photos are blurry, taken at night, or show a vehicle type the AI was never tested on.
• Policy edge cases: the policy started less than 30 days before the accident.
• The customer asks for a person: always honour this.

Here is the routing rule in Python. It covers the limits, low confidence, fraud signs and new-policy triggers; the others add more if lines in the same way. The claim and assessment objects are made up, and assessment holds the AI's payout, score, suggested decision and fraud flags.

AUTO_LIMIT = 15_000        # made-up rupee limits
SENIOR_LIMIT = 150_000

def route_claim(claim, assessment):
    reasons = []
    if assessment.fraud_flags:
        reasons.append("fraud signs: " + ", ".join(assessment.fraud_flags))
    if assessment.decision == "reject":
        reasons.append("AI suggests rejecting")
    if assessment.confidence < 0.90:
        reasons.append(f"low confidence {assessment.confidence:.2f}")
    if claim.policy_age_days < 30:
        reasons.append("policy is under 30 days old")
    if assessment.payout > SENIOR_LIMIT:
        return "senior_review", reasons + ["payout over senior limit"]
    if assessment.payout > AUTO_LIMIT:
        reasons.append("payout over auto limit")
    return ("officer_review", reasons) if reasons else ("auto_pay", [])

The function returns where the claim goes and every reason why. To check it on the bumper claim, save it in a file, add these lines below it, and run the file with Python 3. SimpleNamespace, from Python's standard library, just makes a quick object with named fields.

from types import SimpleNamespace

claim = SimpleNamespace(policy_age_days=400)
assessment = SimpleNamespace(payout=33_000, confidence=0.94,
                             decision="pay", fraud_flags=[])
print(route_claim(claim, assessment))

It prints ('officer_review', ['payout over auto limit']). Change the payout to 9_000 and it prints ('auto_pay', []). Then change the confidence to 0.85 as well, and the claim goes back to an officer with the reason low confidence 0.85.

The two limits are plain numbers at the top of the code, so anyone can read them and a change shows up in code review. The function collects every reason, not just the first. A claim with a low score and a new policy shows the officer both problems.

Very large payouts return early with senior_review, which means two people. Otherwise any reason sends the claim to an officer, and only a claim with zero reasons is paid automatically. A payout of exactly Rs 15,000 is auto-paid, because the test is greater than.

A reviewer screen should show why the case was escalated, then the evidence, then the AI's proposal, then the decision buttons. The order matters. Research on automation bias lists where the advice sits on screen as one design factor that changes how much people lean on it.

Below is the screen the engineer built for the officers, drawn as panels in the order the eye should travel. Notice where the AI's proposal sits: after the evidence, not before it.

The payout is shown line by line, not as one total. The officer sees "Bumper: Rs 21,000, photo 2" and checks that line alone. A single number like Rs 33,000 invites a single click.

The screen also says what the AI could not see. If no photo shows the area under the bumper, the screen says so. Gaps in the evidence are exactly what a person is there to notice.

The officer has four choices, not two: approve, edit the amount, reject, or ask the customer for more photos. An edit or a rejection needs a reason code (a short label from a fixed list), such as "labour hours too high" or "damage looks old". Those codes become very useful later.

That is the core of human review, and beginners can stop here. The rest covers what engineers most often get wrong: whether the AI's score means anything, whether reviewers really check, how many reviewers you need and what to record.

Compare the score with what officers decided on past claims. Group reviewed claims into score bands and count how often the officer accepted the AI's payout in each band. If a band above your threshold is often overruled, the threshold is too low.

This matters because scores are often not honest probabilities. A 2017 study tested modern neural networks, a common kind of model, trained to sort images and documents into classes. It found them poorly calibrated: their scores did not match how often they were actually right. Whatever produces your AI's score, treat it as a ranking to test, not a promise.

def agreement_by_band(history, edges=(0.0, 0.80, 0.90, 0.95, 1.01)):
    """history: (AI score, did the officer accept the AI's payout?) pairs."""
    for lo, hi in zip(edges, edges[1:]):
        band = [agreed for score, agreed in history if lo <= score < hi]
        if band:
            rate = sum(band) / len(band)
            print(f"{lo:.2f} to {min(hi, 1.0):.2f}: {len(band)} reviewed, "
                  f"officer agreed {rate:.0%}")

# Ten made-up reviewed claims. A real check needs far more.
history = [(0.97, True), (0.96, True), (0.96, True), (0.95, False),
           (0.93, True), (0.92, False), (0.91, False),
           (0.88, False), (0.85, True), (0.70, False)]
agreement_by_band(history)

Run it with Python 3. It prints one line per band, with the share of claims where the officer accepted the AI's payout. Each band includes its lower edge but not its upper one.

• 0.00 to 0.80: 1 reviewed, officer agreed 0%
• 0.80 to 0.90: 2 reviewed, officer agreed 50%
• 0.90 to 0.95: 3 reviewed, officer agreed 33%
• 0.95 to 1.00: 4 reviewed, officer agreed 75%

What just happened? In this tiny made-up sample, officers overruled two of the three claims scored from 0.90 to just under 0.95. A 0.90 threshold would have paid all three without review.

Even the top band agreed on only 3 of 4. In a band of four claims, one decision moves the rate by 25 points, and in a band of two by 50. A real check needs many more reviewed claims before you move a threshold.

You stop rubber-stamping by measuring whether reviewers catch mistakes, and by designing the screen and workload so that thinking is easier than clicking. A reviewer who approves everything adds delay but no safety.

Rubber-stamping means approving without really checking, like a clerk who stamps every form. The usual cause is automation bias, the tendency to rely too much on what an automated system says. A 2012 systematic review of 74 papers, focused on healthcare, names workload, task complexity and time pressure as factors that strain attention and feed it.

Here is how it showed up, using made-up numbers. One month after launch, officers approved 97 in 100 AI proposals unchanged, in about 45 seconds each. Pause and predict: if 20 claims with planted mistakes were mixed into one week's queue, how many would they catch?

They caught 6 of 20. This was not laziness. Officers were pushed to clear the queue by evening, and the AI was right so often that checking felt pointless. That is the setting the review describes, so the fix had to be in the system, not in a lecture.

1. 1

   Seed known-answer cases

   Keep mixing a few claims with planted mistakes, called seeded cases, into the queue. The catch rate tells you if review is real. Share it with reviewers as feedback, not as blame.

2. 2

   Ask for a first view before showing the AI's

   For the riskiest claims, the officer enters a rough amount before the AI's proposal appears. Researchers call this a cognitive forcing function, a design that makes you think before you see the answer.

3. 3

   Use forcing where it matters most

   In a 2021 experiment with 199 people, cognitive forcing cut over-reliance more than simple explanations did. People rated the designs that cut over-reliance most the least favourably, so keep them for the riskiest band.

4. 4

   Show evidence, not just a verdict

   The line-by-line payout and the trigger reason give the officer specific things to check. The 2012 review lists whether a system gives information or a bare recommendation among the design choices that affect automation bias.

5. 5

   Watch the numbers per reviewer

   Track approval rate, time per case and catch rate on seeded cases. An approval rate near 100 in 100, or a time far below normal, is a signal to look closer.

6. 6

   Keep the load human-sized

   Cap cases per hour and add short breaks. Workload and time pressure are among the factors the review says strain attention and feed automation bias.

7. 7

   Make the decision visibly theirs

   Put the reviewer's name on every approval and teach the AI's known weak spots, such as night photos. The review lists stressing accountability and training among the ways to reduce automation bias.

Six weeks later, again with made-up numbers, the catch rate on seeded cases rose to 16 of 20. Time per case went up to about 3 minutes. That is slower than 45 seconds and much faster than the old 12 minutes.

What just happened? The approval rate of 97 in 100 looked healthy on its own. Only the seeded cases showed that the review was not working, and only they showed when it started working.

Sort the queue by urgency and waiting time, size the team from the expected number of escalations, and never let a missed deadline approve a case. An SLA (service level agreement) is a promise about speed, such as 90 in 100 escalated claims decided within one working day.

Do the capacity sum before launch, using made-up numbers. 300 claims a day with 40 in 100 escalated gives 120 reviews. At 3 minutes each, that is 6 hours of focused review.

If one officer manages about 4 focused review hours a day, you need two officers on review. Check the sum the other way: two officers give 480 minutes, or 160 reviews of 3 minutes. That leaves room for 40 more a day before the queue starts to grow.

Order matters inside the queue. A customer whose car is stuck at a garage goes first. Two-person reviews come next, because they need two people's time. Then everything else, oldest first, so nothing waits forever.

Turning promises like these into proper SLAs, with targets and alerts, is a topic of its own. The guide on turning AI trade-offs into SLAs covers it.

Record every reviewer decision with its reason code, then study the overrides to fix the AI and tune your triggers. The same record, kept where nobody can edit it, is your audit trail.

An override is any time the reviewer changes the AI's proposal. Each week the engineer groups overrides by reason code. In one made-up week, 30 in 100 overrides said paint cost too high. The cause was the price lookup treating every paint job as metallic paint, which needed one fix, not a new model.

Overrides also tell you when a trigger is too strict. Say officers accept the AI's payout on almost every bumper claim between Rs 15,000 and Rs 25,000 for three months. If the catch rate on seeded cases also stays high, the team might raise the auto-pay limit a little. Move limits in small steps, keep sampling, and write down why.

Reviewers make mistakes too. A second person should recheck a small sample of human decisions, or you may teach the system one reviewer's bad habit. How to turn these signals into better prompts and test sets is covered in the guide on designing feedback loops for LLM systems.

What each audit entry records

Each entry should let a stranger rebuild the decision months later. Here is the team's record in Python. The helpers hash_file, now_utc and audit_log stand in for your own code, and hash_file turns a photo into a fingerprint that changes if the photo changes.

def record_decision(claim, assessment, route, reasons, review=None):
    if review:
        final = review.final_payout
    elif route == "auto_pay":
        final = assessment.payout
    else:
        final = None            # still waiting for a person
    entry = {
        "claim_id": claim.id,
        "photo_hashes": [hash_file(p) for p in claim.photos],
        "model_version": assessment.model_version,
        "prompt_version": assessment.prompt_version,
        "ai_payout": assessment.payout,
        "ai_confidence": assessment.confidence,
        "route": route,
        "route_reasons": reasons,
        "reviewer_id": review.reviewer_id if review else None,
        "final_payout": final,
        "reason_code": review.reason_code if review else None,
        "seconds_on_screen": review.seconds_on_screen if review else None,
        "recorded_at": now_utc(),
    }
    audit_log.append(entry)   # append only: never update or delete

The photo hashes show which photos the decision was based on, even if someone uploads new ones later. Storing the model and prompt versions tells you which AI setup made the proposal, so a bad week can be traced to a change.

Next come what the AI proposed, where the router sent it and why. A reviewed claim adds who decided, the final amount, the reason code and the time on screen. An auto-paid claim gets the AI's payout as final, and a claim still in the queue gets none.

The comment on the last line is only a promise, because code cannot stop someone with database rights from editing old rows. PostgreSQL is a widely used open-source database. In it, give the app's account the INSERT and SELECT rights on the audit table (add and read rows), but not UPDATE or DELETE (change or remove them). Keep approvals in a separate audit table that only the reviewer screen's account can insert into, so the AI side cannot forge one.

Also keep both accounts from owning these tables or being members of the role that owns them. PostgreSQL's documentation says an owner can always grant its own rights back, and members of the owning role can inherit the right to change or drop the table. Superusers can bypass all of this, so keep that list short.

The diagram below puts the whole system together. Follow the main path from claim to payout. Then notice that the router, the officer and the payment all write to the audit log, and that the log feeds the weekly review.

Explain it as a choice you make per action: what the AI may do alone, what a person watches, and what a person must approve first. Then treat the human as a part of the system with limits, costs and failure modes, like a database. Interviewers want to see that you placed people on purpose and can prove the review works.

1. 1

   Classify the actions

   List what the AI can do and score each action on harm, money, reversibility and measured accuracy. Example: pay a small claim, pay a large claim, reject a claim.

2. 2

   Pick a level for each

   Out of the loop for harmless jobs, on the loop with sampling for small reversible ones, and in the loop for the rest. Say that rejections always go to a person.

3. 3

   Write the triggers as code

   Name the limits, the score threshold and the fraud signs. Say how you would check the score against past human decisions, band by band.

4. 4

   Design the reviewer side

   Describe the screen order, the reason codes, the queue priority and the capacity sum for how many reviewers you need.

5. 5

   Close the loop

   Add the insert-only audit log, seeded cases against rubber-stamping, and a weekly review of overrides to tune the limits.