LLM Customer Support Bot System Design: Ticket to Handoff

Urgent problems wait behind repeat questions, and one fault arrives as hundreds of separate tickets. This guide builds an LLM customer support bot system design for exactly that problem.

Picture a common setup. A home broadband provider in Pune has about 1.5 lakh (150,000) customers and 40 support agents across two shifts. The setup is made up, but the problem is real.

On a normal day, customers raise about 4,000 tickets on WhatsApp, web chat, email and the app. A ticket is one customer problem, tracked until it is closed. Each costs about Rs 60 in agent time.

Then, one wet July evening, a digging crew cuts a fibre cable in Kothrud. In five hours, 2,600 customers write in, almost all saying the same thing: no internet. Next morning, the team reads the backlog and finds three patterns.

• Some customers reported the same fault on three or four channels. Agents answered every copy separately.
• A customer whose router had lost its settings waited 20 hours behind the outage tickets. A two-minute fix sat at the back of the line.
• On normal days, about 6 in 10 tickets are the same five questions: no internet, a high bill, slow speed, a plan change and the Wi-Fi password.

The first idea in the room is a chat window connected straight to a large model. That naive version fails three ways. It does not know this provider's plans or prices, so it guesses.

It cannot see a customer's bill, so it cannot explain one. And nothing stops it from chatting on with a customer who wants a person. The design below fixes each failure with a part you can point to.

An LLM customer support bot is a program that reads customer messages, works out what they need, and replies using your help articles and account data. When it should not answer, it hands the ticket to a person.

This guide calls the large language model simply the model. On its own, it knows nothing about the provider's plans, prices or outages. Six more terms carry the rest of the guide.

Intent. The intent is what the customer wants, written as one label. Example: a message about a bill that jumped by Rs 400 gets the intent billing_query.

Retrieval. Retrieval means searching your own documents and pasting the best matches into the prompt, the text your program sends to the model. It is the core of RAG, short for retrieval-augmented generation.

The 2020 paper that named RAG said that showing where a model's answer came from, and updating its knowledge, were still open research problems. Retrieval from your own articles helps with both: you can show which article an answer used, and update an article without retraining anything.

Tool. A tool is a function your code offers the model, such as get_recent_bills or restart_router. The model asks for it. Your code checks the request, runs it and hands back the result.

Handoff. A handoff moves a ticket from the bot to a human agent, along with a short note of everything known so far.

Containment rate. This is the share of tickets the bot closes with no human help. This guide counts only tickets that stay closed, because a ticket reopened days later was never solved.

SLA. A service level agreement is a promise to customers with a number, a time limit and a consequence if it is missed, such as a bill credit. A target the team sets for itself, often stricter, is an SLO, a service level objective.

A good design is a pipeline with clear stages: intake, a queue, triage, an answer step with retrieval and tools, and a handoff path to people.

Follow a ticket from the customer channels in the diagram below. Notice that models work in only two boxes: a small classifier in triage and the large model in the answer worker.

Three choices shape everything else. First, every channel feeds one intake service, so a ticket looks the same whether it came from WhatsApp or email. Second, a queue between intake and the model lets an outage spike wait in line instead of crashing the bot.

Third, the order of steps is fixed in code. This bot is a workflow, where the model works only inside set steps; see our guide on AI agents versus workflows for when a model needs more freedom.

Every stage also writes a log line: ticket id, label, articles used, tools called, cost and time. Without it, nobody can explain a bad reply later.

It sorts each ticket in two passes. Plain code first cleans the ticket up and joins duplicates, then a classifier labels its intent, urgency and confidence, and rules in code pick the route.

Intake: one ticket format for every channel

WhatsApp messages, emails and chat messages all look different. The intake service turns each one into the same record: customer id, channel, text, attachments and time. It also works out who the customer is.

Identity comes from the channel, never from the message text. On WhatsApp, intake matches the registered phone number. On web chat and the app, the customer is already logged in. An email gets a one-time code before the bot shows any account detail.

A phone number proves less than a login. So a WhatsApp chat can read the account, but a plan change or anything that moves money needs a one-time code first. The code goes by SMS to the registered number, or the customer approves in the logged-in app. Treat an account number typed in a message as a claim, not a fact.

Intake also joins duplicates. If a customer writes on WhatsApp and then emails, both messages attach to one open ticket.

Triage: intent, urgency and confidence

One everyday picture helps with three parts of this design: the counter at a busy LPG gas agency. Here is how its parts map to the bot.

• You book a refill from your registered mobile number, and that number finds your connection. That is identity from the channel, not from what you say.
• The counter person fixes nothing. They decide who goes first: a household with no gas left beats one booking early. That is triage.
• A service man going out gets a written slip with the address, the problem and what was tried. That is the handoff note.

Where the picture breaks: a counter person notices when something feels off. A classifier can be confidently wrong, so rules in code back it up, and someone checks its scores.

Ticket triage does the counter's job. The classifier returns three things: the intent (no_internet), the urgency (high after a day offline) and a confidence score from 0 to 1.

The team trains a small, cheap classifier on 50,000 past tickets that agents already labelled. Then it checks that the scores are honest: among test tickets scored about 0.8, about 8 in 10 labels should be right.

A broadband company has one more input: the network team's outage feed, which lists known faults by area. A no-internet ticket from such an area needs no diagnosis.

Here is the decision for one ticket. Follow it from the top. Every question box is a rule in code, not a judgement by the model.

Here is the same logic in Python. The label comes from the classifier, and outage_for_area stands in for a call to the outage feed.

from dataclasses import dataclass

HUMAN_INTENTS = {"cancel_connection", "legal_complaint", "relocation"}
OUTAGE_INTENTS = {"no_internet", "slow_speed"}
OUTAGE_AREAS = {"Kothrud"}  # stand-in for the outage feed

@dataclass
class Ticket:
    area: str
    asked_for_human: bool = False
    bot_attempts: int = 0

@dataclass
class Label:
    intent: str
    confidence: float

def outage_for_area(area):
    return area in OUTAGE_AREAS

def route_ticket(ticket, label):
    if ticket.asked_for_human or label.intent in HUMAN_INTENTS:
        return "human"
    if ticket.bot_attempts >= 2:
        return "human"
    if label.confidence < 0.80:
        return "human"
    if label.intent in OUTAGE_INTENTS and outage_for_area(ticket.area):
        return "outage_update"
    return "bot"

print(route_ticket(Ticket("Kothrud"), Label("no_internet", 0.93)))
print(route_ticket(Ticket("Baner"), Label("no_internet", 0.93)))
print(route_ticket(Ticket("Kothrud", asked_for_human=True), Label("no_internet", 0.97)))
print(route_ticket(Ticket("Baner"), Label("billing_query", 0.62)))

Run it (tested on Python 3.13) and it prints four lines: outage_update, bot, human and human. The third ticket scored 0.97, yet it still goes to a person, because the customer asked for one.

The outage line saves the most effort. On the Kothrud night, about 2,300 of the 2,600 tickets would have got an instant update, all linked to one incident that agents track once.

It combines two kinds of facts: general answers retrieved from help articles, and the customer's own data read through tools. The model is told to write the reply only from those facts. It can still add a detail that is not in them, so the test set checks replies for exactly that.

Retrieval over the help centre

The help centre has about 180 articles on router lights, plan prices, bill dates and more. The team splits them into chunks, small passages of a few paragraphs, searchable by meaning and by keyword.

Each chunk carries two extra fields: the plans it applies to and the date it became valid. Search filters on both in code. A customer on the 100 Mbps plan never gets the 300 Mbps steps, and an old price never beats the new one.

Edited articles are indexed again within ten minutes. If no chunk scores above a set floor, or the model replies with a fixed NO_ANSWER marker, code hands the ticket off. Neither check catches every miss, so missed handoffs are tracked after launch.

Account tools

The help centre explains billing in general. Only the account explains why this customer's bill went up. So the bot gets a few narrow tools.

• get_plan reads the customer's plan, speed and price.
• get_recent_bills reads the last three bills, line by line.
• check_line runs a remote test on the router and fibre line, and changes nothing.
• restart_router restarts the router remotely. The risk is low, and the customer confirms first.
• book_technician offers three free visit slots, and the customer picks one.
• add_credit gives an outage credit of up to Rs 200, worked out in code from outage hours, after a one-time code for WhatsApp or email chats. Anything larger goes to a person.

Every tool takes the customer id from the verified session, never from the model. If a message mentions another account number, the tools ignore it.

OWASP, a non-profit that publishes security guidance, calls this risk excessive agency: a model with more functions, permissions or freedom than its job needs. Its 2025 list advises the fewest tools, the least access, and acting as the specific user.

It also advises that a person approve high-impact actions. Its 2025 entry on sensitive information disclosure warns that limits written in the system prompt can be bypassed.

The system prompt is the fixed instructions the bot gets before every chat. Prompt injection, text crafted to steer the model, can talk past those instructions. So this bot's limits live inside the tools.

A support bot should hand over when a rule says so, when it is unsure, or when it has tried and failed. The handoff should carry everything the agent needs, so the customer never repeats the problem.

• The customer asks for a person, in any words.
• The intent is on the always-human list: cancellation, legal complaints and moving house.
• The classifier's confidence is below 0.80, or the help centre has no matching article.
• Two bot replies have not fixed it, or the customer sounds angry or upset.
• An action is above a limit, such as a credit over Rs 200.

The handoff note is the gas agency's written slip. It holds the intent, a three-line summary, the account facts already fetched, the line test result and every reply the bot sent. The answer worker writes the summary, or the triage worker makes one short model call when it hands off directly.

AI can help the agents on the receiving end too. A 2023 study of 5,172 support agents found that an AI assistant raised issues resolved per hour by 15% on average.

The gains were uneven. Less experienced and lower-skilled agents improved in both speed and quality, while the most experienced saw small speed gains and small declines in quality.

Approval screens, queue design and escalation have more depth than fits here. For that, see our guide on human-in-the-loop escalation and approvals.

In this example, a ticket the bot closes costs about Rs 1, while a person costs Rs 45 to Rs 60 per ticket. The number that matters is cost per resolved ticket, counting tickets that come back.

Model services usually charge by tokens, small pieces of text that are often a word or part of a word. You pay for tokens going in and coming out. So cost depends on how long the chat runs and how many articles go into each prompt.

The team's estimate for one bot ticket has three parts. Triage costs Rs 0.10. The answer step averages three model calls, about Rs 0.60 in total. Search, tools and servers add Rs 0.30.

A handed-off ticket costs Rs 45 instead of Rs 60, because the note saves the agent's first few minutes. A reopened ticket costs the full Rs 60, since the agent must undo a wrong answer.

Daily cost falls from Rs 2,40,000 to Rs 1,10,200, that is from 2.4 lakh to about 1.1 lakh rupees, a 54% saving. To check it, multiply each row's tickets by its cost and add the three bot rows.

Two lessons sit in this table. The bot's own cost is tiny next to people's time, so never cut quality to save paise (hundredths of a rupee). And a wrong close costs Rs 61 against Rs 1 for a right one, so watch the reopen row closely.

On outage days, updates skip the answer step, so each Kothrud ticket would cost about Rs 0.40. The team also uses these controls to keep costs steady.

• Cap each chat at a fixed number of bot turns before handoff.
• Send only the top three chunks into the prompt.
• Use the small classifier for triage, and keep the large model for replies.
• Log cost per ticket by intent, so a slow rise shows up within a week.

A support bot needs targets for how fast it first replies, how fast a person picks up a handoff, and how often it is available. Each target needs a number, a time window and a plan for misses.

Speed targets work best on a percentile. The 95th percentile, or p95, is the time that 95 out of 100 replies meet or beat. Google's Site Reliability Engineering (SRE) book warns that an average can hide this slow tail.

These are the team's internal targets, its SLOs:

• Bot first reply on chat, WhatsApp and the app: within 10 seconds at p95.
• Bot first reply on email: within 5 minutes at p95.
• Human pickup after a handoff at p95: 15 minutes for urgent tickets, 4 hours for the rest, in working hours.
• Outage update: within 1 minute of the network team marking an outage.
• Bot availability: 99.5% of each month, which allows about 3.6 hours of downtime in a 30-day month.

The availability target needs a fallback. If the model service is down, intake still works: the customer gets an acknowledgement with a ticket number, and the ticket goes to the human queue.

Targets pull against each other. A bigger model may write better replies but miss the 10-second target, and more agents cost more per ticket. For turning trade-offs into promises, see our guide on AI SLAs.

Group held-out test tickets by the score the classifier gave, then compare each group's score with how often its labels were right. Held-out tickets are labelled tickets the classifier never trained on.

A classifier whose scores match its hit rate is called calibrated. A 2017 paper found that modern neural networks, unlike those from a decade earlier, are poorly calibrated. So never assume a classifier's scores are honest: measure the match, because the 0.80 rule is only as good as this match.

Here is the check in plain Python, on 100 made-up test results. Swap in your own pairs of score and right-or-wrong.

# Each pair: (score the classifier gave, was its label right?)
# 100 made-up held-out tickets. Use a few hundred real ones per bucket.
results = (
    [(0.95, True)] * 46 + [(0.95, False)] * 4
    + [(0.85, True)] * 21 + [(0.85, False)] * 9
    + [(0.75, True)] * 12 + [(0.75, False)] * 8
)

buckets = {"0.7 to 0.8": (0.7, 0.8), "0.8 to 0.9": (0.8, 0.9), "0.9 to 1.0": (0.9, 1.01)}
for name, (lo, hi) in buckets.items():
    hits = [ok for score, ok in results if lo <= score < hi]
    print(f"scored {name}: {len(hits)} tickets, {sum(hits) / len(hits):.0%} right")

Run it and it prints three lines: 60% right for scores 0.7 to 0.8, 70% for 0.8 to 0.9, and 92% for 0.9 to 1.0.

The middle line is the problem. Those tickets pass the 0.80 rule, yet 3 in 10 of their labels are wrong. Either raise the threshold to 0.90 or recalibrate the scores, then run the check again.

The same paper found that temperature scaling, a fix that divides the model's raw output scores by one number tuned on held-out data, worked well in its tests. With only 30 tickets in a bucket, one wrong label moves that bucket's result by about 3 points, so use a few hundred.

Test it on hundreds of real past tickets before launch, run each test several times, then roll it out one ticket type at a time. Keep a way to switch the bot off at every stage.

A test set from real tickets

The team picks 600 past tickets across all intents, messy ones included. For each, it writes the right route and, for bot tickets, what a correct reply must contain. The set runs before every change.

Why one run per test is not enough

A model can answer the same message differently each time. A 2024 benchmark paper tested agents in simulated customer conversations, where each agent had tools and policy rules, in domains such as retail.

Its authors report that state-of-the-art agents of the time succeeded on fewer than half of its tasks. They also proposed pass^k, the chance that an agent solves the same task in all k tries.

In retail, pass^8 was under 25% even for those state-of-the-art agents. So the team counts a test ticket as passed only if it passes all four runs.

The code below uses a stand-in bot that is right on 90% of runs, so you can run it anywhere. Swap flaky_bot for your real bot's handler.

import random

random.seed(42)

def flaky_bot(text):
    # Stand-in for the real bot: right on 90% of runs
    if random.random() < 0.9:
        return "bot", "The Wi-Fi password is printed on the router label."
    return "human", ""

def passes_every_time(handle, case, runs=4):
    for _ in range(runs):
        route, reply = handle(case["text"])
        if route != case["expected_route"]:
            return False
        if route == "bot" and case["must_contain"] not in reply:
            return False
    return True

case = {"text": "wifi password?", "expected_route": "bot", "must_contain": "router label"}
cases = [case] * 600
once = sum(passes_every_time(flaky_bot, c, runs=1) for c in cases)
every = sum(passes_every_time(flaky_bot, c) for c in cases)
print(f"pass one run: {once} of 600")
print(f"pass all four runs: {every} of 600")
print(f"maths says: {0.9 * 600:.0f} and {0.9 ** 4 * 600:.0f}")

Run it and it prints 541 of 600 for one run and 395 of 600 for all four runs. The maths line prints 540 and 394, because 0.9 to the power 4 is about 0.66.

Each run is a fresh chance to fail, so the chances multiply. Even a bot right on 99% of runs passes all four only about 96% of the time.

The formula holds for one ticket whose runs fail independently. Across a test set, some tickets always pass and some always fail, so measure the pass-all-runs rate directly. In the code, the route check comes first, because a wrong route is the costliest mistake.

Numbers to watch after launch

• Containment rate, counting only tickets not reopened within 7 days.
• Reopen rate for bot-closed tickets, for each intent.
• Missed handoffs: bot-closed tickets that agents later mark as needing a person.
• Customer rating after each bot chat.
• p95 reply time and cost per resolved ticket, for each intent.

Rolling out in stages

Here is the rollout plan. Read it from the top, and notice the arrow that switches an intent off.

In the first stage, the bot only drafts replies, and agents check and send them. Shadow mode then runs the bot on live tickets without sending anything, so its routes can be compared with what agents did.

The pilot starts with two safe intents, outage updates and Wi-Fi password help, for 10% of customers. A settings switch turns any intent off in seconds and sends its tickets back to people.

Walk the interviewer through one ticket's journey, with a number at each stage.

1. 1

   Ask about scale and channels

   Ask how many tickets arrive a day, on which channels, and what a spike looks like. For the broadband example: 4,000 a day, and 2,600 in five hours in an outage.

2. 2

   Draw intake and a queue

   One intake service gives every channel one ticket format and checks identity from the channel. A queue absorbs spikes before any model runs.

3. 3

   Add triage and routing rules

   A small classifier labels intent, urgency and confidence. Plain code picks the route: human rules, confidence, outage check, then the bot. Say how you check the scores are calibrated.

4. 4

   Design the answer step

   Retrieval over dated, plan-tagged articles, plus narrow account tools that use the verified customer id and enforce money limits.

5. 5

   Design the handoff

   List the triggers, what the note carries, and how fast a person picks it up.

6. 6

   Do the cost and SLA maths

   Estimate cost per resolved ticket, counting reopens. State the p95 reply target and the fallback when the model is down.

7. 7

   Close with testing and rollout

   A test set of real tickets, each passing every one of several runs, then agent assist, shadow mode and a staged pilot with a stop rule.