AI Incident Response: When Your AI Gets It Wrong in Public

The company still owns every answer the bot gives, so wrong answers turn into angry customers, refund claims and screenshots online. Worse, nothing looks broken while it happens. The servers are up, and the bot is simply wrong.

Picture a common setup. An intercity bus ticket app sells seats on routes across Andhra Pradesh and Odisha. Its chatbot answers questions about routes, boarding points and refunds. The setup is made up, but the problem is real.

The real refund rule is simple. Cancel more than 24 hours before departure and you get 90 percent back. Cancel between 6 and 24 hours before, and you get half. Inside 6 hours, you get nothing.

On a Friday at 4 pm, the marketing team adds a new document to the bot's knowledge store, the set of pages the bot reads before it answers. The document describes a Diwali offer: free cancellation on three routes, for one week only.

The bot misses the fine print. From about 7 pm it tells people on every route that they can cancel right up to departure for a full refund.

• Saturday, 9 am: the support lead sees 40 tickets where customers quote the bot's promise of a full refund.
• Saturday, 10 am: a customer posts a screenshot of the chat, and it spreads across local social media groups.
• Saturday, 10:20 am: the support lead calls the on-call engineer. On-call means the person who must respond to problems that week.
• Saturday, 11 am: two local reporters call the communications lead to ask whether the refunds are real.
• Saturday, 1 pm: the bot is finally off, after a code change and a 40-minute release.

The on-call engineer's first move was the naive one: rewrite the bot's instructions and hope. That took an hour and did not stop the wrong answers. The better order is to stop the damage first, then fix it calmly.

You need six words: incident, kill switch, fallback, rollback, incident commander and post-mortem. Each gets a plain meaning and an example from the bus app.

Incident. An incident is any unplanned event that harms users or the business and needs a response now. Example: the bot promising full refunds the company never offered.

Kill switch. A kill switch is a setting that turns a feature off, or limits it, without a new release. Example: one switch that stops the bot answering refund questions.

Fallback. A fallback is the safe thing users get while the feature is off. Example: a fixed message that points to the exact refund on the customer's booking page, plus a button to reach a person.

Rollback. A rollback puts back the last version that worked. Example: removing the Diwali document from the knowledge store and returning to Thursday's set of pages.

Incident commander. The incident commander is the one person who runs the response and makes the calls while others do the work. Example: the on-call engineer leads, the support lead runs the queues and the communications lead writes the messages.

Post-mortem. A post-mortem is a written review after the incident: what happened, why, and what will change. Blameless means it looks for gaps in the system, not a person to punish.

AI incident response is the plan and the practice for what your team does when an AI feature causes harm in real use. You stop the harm, tell the people affected, fix the cause, and learn from it.

The idea comes from security and operations teams. NIST, the US government's standards agency, revised its incident response guide as SP 800-61 Revision 3 in April 2025. It says incident response is a critical part of cybersecurity risk management.

NIST's AI Risk Management Framework (AI RMF), a voluntary framework released in January 2023, carries the same idea over to AI. Its MANAGE 2.4 item asks for mechanisms, with assigned owners, to replace, disengage or switch off an AI system whose results do not match its intended use.

Its MANAGE 4.3 item asks that incidents and errors are communicated to the relevant people, including affected communities. It also asks that tracking, response and recovery follow documented processes. As of 19 October 2026, NIST's page says AI RMF 1.0 is being revised.

Think of a floodlight failure during a night match at a cricket stadium. The umpires stop play at once, before anyone gets hurt chasing a ball they cannot see. The announcer tells the crowd what happened and what comes next. Later, the board holds an inquiry.

Map it part by part. Stopping play is your kill switch. The announcement is your communication. The inquiry is your post-mortem. Where the picture breaks: a floodlight failure is loud and everyone sees it, while a wrong answer is quiet, so you must build the eyes first.

The diagram shows the full cycle in order. Notice that the fix comes only after the harm has stopped, and that the last step feeds back into detection.

A normal outage is usually loud: the app is down and dashboards turn red. An AI incident is often quiet: the system is up, fast and confidently wrong.

During the bus app's incident, every server health check passed. Replies came back in under two seconds. The only sign of trouble was in the words the bot wrote, and no alert was reading those.

The trigger row matters most. The team changed no code that Friday; the change was a single document. In an AI feature, a document or an instruction can change answers as much as code can, so it needs the same testing.

An AI kill switch is a setting that turns an AI feature off, or limits it, without a new release. Your code checks it on every request, so a flip takes effect within seconds. If servers keep a cached copy of the switch, add that cache time.

Software teams call these ops toggles. Pete Hodgson's 2017 article on feature toggles, published on Martin Fowler's site, says they let operators disable or degrade a feature quickly in production. It even calls the long-lived ones kill switches.

The article adds that ops toggles must be changed very quickly, and that needing a new release to flip one is unlikely to make operations staff happy. The bus app learnt this the hard way. Its only switch was a line in the code, so turning the bot off took a 40-minute release.

A good kill switch has levels, so you can turn off the smallest part that fixes the problem:

• Whole bot off. Every chat goes to the human support queue. Use it when you cannot yet tell what is wrong.
• One topic off. The bot still answers about routes and boarding points, but refund questions get a fixed, correct message.
• One source off. A single document or tool is hidden from the bot, such as the Diwali offer, while everything else keeps working.
• One action off. If the bot can do things, like issue a refund, that action is blocked while answers continue.

The diagram shows the bus app's new setup. Start at the switch panel. Follow how one flip changes the path of the very next chat.

You know through signals you set up in advance: automatic checks on what the bot says, sudden jumps in support tickets, and a record of every recent change. Without them, customers find the problem first.

In the bus app, customers found it first. The wrong answers started at about 7 pm on Friday, and support saw the pattern at 9 am on Saturday. That gap of 14 hours is the time to detect, and shrinking it is most of the battle.

After the incident, the on-call engineer added these signals:

• A promise counter. Code counts replies that mention a refund amount or the words full refund, per hour. A sudden jump raises an alert.
• A fact check on money answers. Any reply that states a refund is compared with the refund rule in code. A mismatch is logged and alerted.
• A ticket tag. Support staff tag tickets that quote the bot, and 10 or more in an hour pages the on-call engineer.
• A change log. Every new document, instruction change or model update is recorded with a time, so a spike can be matched to a change.

The diagram shows how these signals meet. Follow the arrows into the question box. That is where the on-call engineer decides whether to declare an incident.

Watching AI answers over time is its own topic, often called LLM observability. LLM means large language model, the kind of AI behind most of today's AI chatbots. SevyDevy's LLM observability guide also covers slow drops in quality and creeping costs.

In the first hour, declare the incident, contain the harm with the smallest switch that works, and send a first honest message. Finding the root cause comes later.

Here is how the bus app's first hour would run today, with the new switches in place. The minute marks are made up.

1. 1

   Minute 0: declare

   The on-call engineer opens an incident channel and takes the commander role. They set the severity, meaning how serious it is: high, because money and public trust are involved.

2. 2

   Minute 3: stop play

   The commander switches off the refund topic. Customers now get the fixed refund message. The bot keeps answering route and boarding questions.

3. 3

   Minute 5: check the switch worked

   The commander sends a test refund question and sees the fallback. The promise counter should fall to zero in the next hourly count.

4. 4

   Minute 10: preserve evidence

   The team saves the bot's logs, the current documents and the change log. Nobody deletes or edits anything until the post-mortem.

5. 5

   Minute 15: first message

   The communications lead posts a short note in the app and on social media: refund answers were wrong, they are paused, and an update follows.

6. 6

   Minute 30: find the change

   The change log shows the Diwali document added on Friday at 4 pm. The engineer hides that one source from the bot.

7. 7

   Minute 60: plan the next update

   The support lead adds staff to the queue. The team agrees when the next public update will go out, and who writes it.

Notice what is missing from the first hour: rewriting the bot's instructions. A fix written under pressure, with angry posts arriving, is how one incident becomes two. Fix in a calm hour, test it, then switch the topic back on.

Say quickly and plainly what went wrong, who is affected and what you will do about it. Own the mistake as the company's, never as the bot's.

A real case shows why. Moffatt v. Air Canada (2024 BCCRT 149) was decided on 14 February 2024 by British Columbia's Civil Resolution Tribunal, as a small claims dispute. The airline's chatbot told a passenger he could apply for a bereavement fare after travelling.

Another page on the airline's website said otherwise. The tribunal said the airline was, in effect, suggesting the chatbot was a separate legal entity responsible for its own actions. The tribunal rejected that and held the airline responsible for all the information on its website.

It found the airline did not take reasonable care to make its chatbot accurate. It also asked why customers should have to double-check one part of a website against another. The airline was ordered to pay 812.02 Canadian dollars, including 650.88 in damages.

It is one small-claims decision from one Canadian province, not Indian law. Still, the lesson is plain: the bot is the company speaking, and blaming the bot in public will sound like an excuse.

The communications lead writes three messages, each for a different reader:

• Staff first. Support agents get the facts and a script, so nobody contradicts the public message or promises something new.
• Affected customers next. Each one gets a direct message that says what the bot got wrong and what the company will do for them.
• The public last, but within the hour. A short post in plain words, then updates at the times promised, even when there is little news.

Then comes the hard decision: honour the wrong promises or not. The company's leaders choose to honour them for every booking that received one. It costs money, but they judge that refunds cost less than customers who never book again.

A log scan finds 310 affected bookings (made up), and the engineer part below shows how. The average gap between the promised and the real refund is Rs 640. If every one of them cancels, the bill is Rs 1,98,400, about 2 lakh rupees (1 lakh is 100,000).

Hold a review within a few days and rebuild the timeline from logs. Find why the system allowed the mistake, and give each finding an action with an owner and a date.

This is the inquiry after the floodlight failure. The board does not just ask who flipped which switch. It asks why one fault could darken the ground, and why nobody saw it coming.

PagerDuty, an incident management software company, publishes its own incident response guide. As of 19 October 2026, it asks for a post-mortem after every major incident, meaning its two most severe levels, SEV-1 and SEV-2.

It schedules the meeting within 3 calendar days for a SEV-1 and 5 business days for a SEV-2. It also asks that every timeline entry names the metric or page its data came from.

The bus app's post-mortem document has five parts:

• Summary. Two or three sentences a new employee could follow.
• Impact. How many chats, bookings and rupees, and for how long. Here: about 18 hours from the first wrong answer to the bot going off.
• Timeline. Each event with its time and a link to the log or ticket that proves it.
• Causes. The trigger, plus every gap that let the trigger cause harm.
• Actions. Each with one owner, a due date and a way to check it is done.

The trigger was the Diwali document. But the review asks why one document could change refund answers on every route. The team finds four gaps.

1. 1

   Gap 1: content skipped testing

   New documents went live with no checks. Action: every new document must pass a set of 50 refund test questions before release. Owner: the engineering lead.

2. 2

   Gap 2: no limits on the offer

   The document never named its routes in a field the bot could filter on. Action: offers carry route and date fields. Owner: the marketing lead.

3. 3

   Gap 3: nobody watched the answers

   No alert read what the bot said. Action: the promise counter and the refund fact check. Owner: the engineering lead.

4. 4

   Gap 4: the switch needed a release

   Turning the bot off took a 40-minute release. Action: the flag store with levels, drilled monthly. Owner: the engineering lead, with support checking the fallback.

Notice that no gap reads as someone in marketing making a mistake. The marketing team did its normal job. The system had no step that would catch an ordinary slip, and that is what the actions fix.

One more habit helps. Turn every incident into new test cases. The Diwali chats become test questions that run before every future document or instruction change, so this exact mistake is caught before release.

Run a drill: put the switch into each state, including a broken flag store, and check what a refund question and a route question get. If you have never flipped it, you do not know it works.

A simple first version, like the one an earlier edition of this guide showed, has two quiet bugs. If the flag store throws an error, the whole chat crashes. And if any topic is blocked, it returns the refund message, even for route questions.

The version below fixes both. The helpers load_flags, hand_to_agent, classify_topic, log_event and ask_bot are stand-ins for your own code.

FALLBACKS = {
    "refund": "Refund answers are paused. Your exact refund is on My Trips.",
}
DEFAULT_FALLBACK = "This answer is paused. Tap Talk to a person for help."


def safe_flags():
    try:
        return load_flags()
    except Exception:
        return {}  # store unreachable: treat it as "bot off"


def reply(chat):
    flags = safe_flags()  # read on every chat
    if flags.get("bot_enabled") is not True:
        return hand_to_agent(chat)  # whole bot off
    topic = classify_topic(chat["message"])
    if topic in (flags.get("blocked_topics") or []):
        log_event("topic_blocked", chat["id"], topic)
        return FALLBACKS.get(topic, DEFAULT_FALLBACK)  # one topic off
    return ask_bot(chat)

Two lines matter most. safe_flags turns any flag store error into an empty set of flags. And only a real True turns the bot on, so an empty store, an unreachable store or a typo like the text false sends every chat to people.

The drill below gives each helper a tiny fake version and walks through four switch states. Paste both blocks into one file, save it as drill.py and run python3 drill.py.

# Stand-ins so the drill runs on its own. Put reply() above this in the same file.
STORE = {}


def load_flags():
    if STORE.get("down"):
        raise ConnectionError("flag store unreachable")
    return dict(STORE)


def hand_to_agent(chat): return "HUMAN QUEUE"
def classify_topic(text): return "refund" if "refund" in text else "route"
def log_event(*args): pass
def ask_bot(chat): return "BOT ANSWERS"


refund = {"id": 1, "message": "Can I get a refund?"}
route = {"id": 2, "message": "Which route stops at Vijayawada?"}
drills = [
    ("all on", {"bot_enabled": True}),
    ("refund off", {"bot_enabled": True, "blocked_topics": ["refund"]}),
    ("whole bot off", {"bot_enabled": False}),
    ("store down", {"down": True}),
]
for name, state in drills:
    STORE.clear()
    STORE.update(state)
    print(f"{name}: refund -> {reply(refund).split(".")[0]} | route -> {reply(route)}")

On Python 3.13 it prints these four lines:

all on: refund -> BOT ANSWERS | route -> BOT ANSWERS
refund off: refund -> Refund answers are paused | route -> BOT ANSWERS
whole bot off: refund -> HUMAN QUEUE | route -> HUMAN QUEUE
store down: refund -> HUMAN QUEUE | route -> HUMAN QUEUE

What just happened? With everything on, both questions reach the bot. With the refund topic off, only refunds get the fixed message. With the bot off or the store down, both go to people. A store failure, or a bot_enabled value that is not exactly True, never leaves the bot answering.

Budget for the flag cache

Reading a remote flag store on every chat adds a network call each time. To avoid that, you can cache the flags in each server's memory for a few seconds. The cost is a delay: a flip reaches each server only when its cache refreshes.

We simulated a 30-second cache and flipped the switch at 300 different moments. The worst case was 30.0 seconds before the server saw the change, which is the cache time, as you would expect. So write the cache time into your runbook, the team's written incident steps, and time the real flip in each drill.

Scan the saved chats from the incident window for promise wordings, then have people read anything the pattern cannot judge. A text pattern finds words, not meaning, so it needs a human check on both sides.

Each saved chat here has an hour (counted from when the Diwali document went live), a topic label, the booking and the bot's reply. The window runs from hour 0 to hour 21, when the bot went off at 1 pm.

import re

PROMISE = re.compile(r"full refund|100 ?% refund|no cancellation charge",
                     re.IGNORECASE)
NEGATION = re.compile(r"\b(not|no|never|cannot)\b|n['’]t\b", re.IGNORECASE)


def scan(chats, start, end):
    promised, to_read = set(), set()
    for chat in chats:
        if chat["topic"] != "refund" or not start <= chat["hour"] <= end:
            continue
        if not PROMISE.search(chat["reply"]):
            continue
        rest = PROMISE.sub("", chat["reply"])  # keep "no cancellation charge"
        if NEGATION.search(rest):
            to_read.add(chat["booking"])   # maybe "not a full refund"
        else:
            promised.add(chat["booking"])
    return sorted(promised), sorted(to_read - promised)


chats = [
    {"hour": 3, "topic": "refund", "booking": "B2",
     "reply": "Yes, a FULL REFUND up to departure."},
    {"hour": 4, "topic": "refund", "booking": "B1",
     "reply": "Cancel any time for a 100% refund."},
    {"hour": 5, "topic": "refund", "booking": "B1",
     "reply": "You still get a 100 % refund."},
    {"hour": 6, "topic": "refund", "booking": "B3",
     "reply": "This ticket is not eligible for a full refund."},
    {"hour": 7, "topic": "refund", "booking": "B4",
     "reply": "You will get all your money back."},
    {"hour": 8, "topic": "refund", "booking": "B6",
     "reply": "A full refund is not possible on this ticket."},
    {"hour": 30, "topic": "refund", "booking": "B5",
     "reply": "Full refund on this route."},
]
print(scan(chats, start=0, end=21))

Save it as scan.py and run python3 scan.py. On Python 3.13 it prints one line: (['B1', 'B2'], ['B3', 'B6']).

What just happened? B1 is listed once, though the customer asked twice, because a set keeps one copy of each booking. B2 matched in capital letters, since the pattern ignores case. B5 is outside the window.

B3 and B6 say a full refund is not available. A plain pattern search would count both as promises. The negation check first removes the promise words, so a reply saying no cancellation charge still counts as a promise. Then any reply left with not, no, never, cannot or a word ending in n't goes to people.

B4 promised all the money back in words the pattern never thought of, so it was missed. That is why the support team also reads a random sample of refund chats that did not match, then adds any new wording to the pattern. The negation list is only a starting point, so the sample should include some promised chats too.

Explain it as a cycle of detect, contain, communicate, fix and learn, and show where each step lives in your design.

1. 1

   Name the quiet failure

   Say that an AI feature can stay up and fast while giving wrong answers, so normal health checks are not enough.

2. 2

   Draw the signals

   Add checks on the answers themselves, a ticket signal and a change log. Say what triggers an alert.

3. 3

   Draw the kill switch

   Show a flag store checked on every request, with levels: whole feature, one topic, one source, one action. Say it fails safe when the store is down.

4. 4

   Describe the fallback

   Say what users get while the switch is on, such as a fixed correct message and a human queue.

5. 5

   Cover communication

   Mention one incident commander, messages for staff, affected users and the public, and how you find affected users from logs.

6. 6

   Close with the post-mortem

   Blameless review, owned actions with dates, and new test cases built from the incident's own conversations.