Design an AI Chat Assistant: System Design Walkthrough

In a chat app, the failures people notice usually come from the parts around the model, not the model itself. AI chat assistant system design is the work of planning those parts. Replies feel frozen, one script runs up the bill, and long chats fail.

Picture a common setup. A chat assistant app in India answers questions in English, Hindi and Kannada for about 15 lakh (1.5 million) people a day. The setup is made up, but the problems are real.

On a festival night, traffic triples in two hours. Four things break, and each one points to a part of the design in this guide.

• Replies arrive in one block after up to 20 seconds. Users think the app has frozen and press send again, which doubles the load.
• One account runs a script that sends 40,000 messages in a night. At about Rs 0.14 a message, that costs the team about Rs 5,600.
• Very long chats fail, because the app sends the whole history every time and goes past the model's input limit.
• The model service in one cloud region, a group of data centres in one area, slows down for ten minutes, and every user sees a spinner.

The team now needs a design that can carry 50 lakh daily users next year. This guide builds it the way a good interview answer does: requirements, numbers, then the parts.

AI chat assistant system design is planning the services around a model so that many people can chat with it quickly, safely and at a cost you can afford. The model writes the words. Your system does everything else.

What must the assistant do?

Functional requirements say what the product does. Non-functional requirements say how well it must do it: speed, uptime, safety and cost. Here are the functional ones.

• Users send a message and see the reply appear word by word.
• Users can reopen old chats, continue them and delete them.
• Free and paid plans get different daily limits.
• Users can stop a reply halfway.

These are the non-functional targets the team agrees on. The numbers are made up, but their shape is typical.

• The first words of a reply appear within 1.5 seconds for most messages. This wait is called time to first token.
• At least 99.9% of messages get a reply each month, even if one cloud region fails.
• No reply reaches the screen without a safety check.
• Model cost stays under about Rs 6 per user per month.

A token is a small chunk of text, often a whole word or part of one. Models read and write tokens, and hosted model services usually charge for the tokens going in and coming out.

How big is the load?

Rough numbers, called back-of-the-envelope estimates, show which parts need the most care. Take 50 lakh daily users sending 6 messages each. That is 3 crore (30 million) messages a day, or about 347 a second on average.

Evenings are busier, so the team plans for a peak of about 1,400 messages a second, four times the average. Each reply takes about 8 seconds to stream. So at peak, about 11,200 replies are streaming at the same moment.

Storage is smaller than people expect. Each message saves two rows, the user's text and the reply, at about 1.5 KB each. That is about 90 GB a day, or about 33 TB a year.

Each box below is one part of the system, and the line under its name says what it does. Follow a message from the app to the model pool.

You keep one HTTP response open and send each new piece of text as a small event the moment the model writes it. Server-sent events (SSE) is the web standard built for this one-way flow.

A model writes its reply one token at a time. A 300-token reply might take 8 seconds to finish, yet the first token is ready much sooner. Streaming lets the user start reading at once.

Think of a live radio phone-in show. The listener rings once with a question, and the answer comes over the air as the presenter speaks it. Here is how the parts map.

• The phone call is the app's one request.
• The live broadcast is the open HTTP response.
• Each sentence the presenter speaks is one event.
• The station's few-second broadcast delay is the safety hold-back you will meet later.

Where the picture breaks: a radio listener who tunes in late misses the start for good. An SSE client that drops off can ask for the rest, as the engineer section shows.

The HTML standard defines the format. The server sends the content type text/event-stream. Each event is a few lines, such as id, event and data, ended by a blank line. A line starting with a colon is a comment, which the browser ignores.

The browser's built-in SSE client is called EventSource. Its only option is whether to send credentials such as cookies, so it cannot send a message body. It also cannot add request headers, the labelled lines of extra detail sent with a request.

An app that sends a login token in a header therefore needs another way to authenticate streams, such as cookies. The app first sends the message with a POST request, an HTTP request that carries data. It gets back a message id, then opens the stream for that id.

You can watch a stream yourself with curl, a command-line tool for HTTP requests; the engineer section shows the exact commands. Events should print one at a time, about as fast as the model writes, not all at the end.

This diagram shows one full turn. Each box is one message between two parts, and the small line says who talks to whom.

Store each conversation as one row and each message as its own row, keyed by conversation id and a sequence number. Save the user's message before you call the model, and save the reply when it ends.

The team uses two tables. A conversations table holds the owner, the title and a running summary. A messages table holds one row per message, as shown below.

The status column handles the messy cases. A reply starts as streaming. It becomes complete at the end, stopped if the user tapped stop, or failed if the model broke halfway. A partial reply is still saved, so a chat never shows a hole.

The team shards the data by user id, which means splitting it across many database machines by that key. One user's chats then sit on one machine, so loading their history is one quick query.

Chats untouched for 90 days move to object storage, which is cheap storage for large files. They come back when someone opens them, so the database stays small and fast.

Deletion must be real. When a user deletes a chat, remove its messages, its summary and any archived copies. Backups expire on their own schedule, so state that period in your privacy policy. Logs keep message ids and token counts, never the text.

You give each request a token budget and fill it in order of value: the system prompt, a summary of older turns, the latest turns and the new message.

The system prompt is the fixed instructions sent first on every request. The context window is the most text a model can handle in one request, counted in tokens. The reply is written into that same window, so it needs room too.

Bigger is not always better for quality either. A 2023 study, Lost in the Middle, tested question answering over many documents and a key-value lookup task. Models often did best when the key fact sat near the start or end of a long input, and worst in the middle.

The team sets a budget of 16,000 tokens per request, well below what the model allows, with 1,000 kept free for the reply. To keep requests small, a background worker asks a small, cheap model to fold all but the latest few turns into the running summary. The 16,000 cap then catches only unusual requests.

Picture a 600-message chat about a Goa wedding. The summary keeps a few lines: the date, the budget, the guest count and the user's preferences. Our guide on context engineering for AI agents covers memory and compaction in depth.

Rate limits cap how fast one user can send requests, and quotas cap how much they can use over a day or a month. Together they stop one account from slowing everyone down or running up a huge bill.

OWASP, a non-profit that publishes security guidance, lists unbounded consumption in its 2025 top 10 risks for LLM (large language model) applications. One example is denial of wallet: flooding a pay-per-use service with requests to run up its bill.

OWASP's advice includes rate limits, user quotas, input size limits, timeouts and throttling. OWASP also posted a 2026 edition on 3 August 2026; this guide quotes the 2025 entries, as of October 2026.

The team uses four controls, with made-up numbers.

• Request rate: at most 20 messages a minute per user, to stop scripts and stuck retry loops.
• Open streams: at most 2 replies streaming at once per user, because each stream holds a connection for seconds.
• Token quota: a daily allowance by plan, such as 2 lakh tokens for free users and 20 lakh for paid ones.
• Message size: anything over 8,000 tokens is refused before it reaches the model.

When a limit is hit, the API replies with HTTP status 429, Too Many Requests. RFC 6585, the standard that defines it, says the server may add a Retry-After header saying how long to wait. The app reads that header and shows a countdown.

Count tokens, not just requests, because requests differ hugely in size. One 6,000-token paste carries as many input tokens as 60 short 100-token questions, yet a request counter sees just one. The chat service reserves an estimate before the call, then corrects it with the real count afterwards.

RFC 6585 leaves it to you how to identify the user and how to count. Our guide on rate limiter system design shows how the counters work inside.

Here are three causes we reproduced, all sitting between your server and the user. A proxy can hold the stream back, a proxy can close a quiet connection, and the browser reconnects even after a reply has finished.

Proxies buffer the stream by default

We ran exactly that on nginx 1.24 with a tiny Python SSE server. Direct to the server, events arrived 0.5 seconds apart. Through nginx with default settings, all six events arrived together at 3.0 seconds, when the server finished.

What just happened? The nginx setting proxy_buffering is on by default, so nginx collected the response before passing it on. The header X-Accel-Buffering: no, sent by the chat service, turned buffering off for that one response.

A longer reply does not wait for the end, but it still comes in lumps. Our 12 KB test stream, sent over 5 seconds, arrived in three bursts about 1.7 seconds apart.

Other proxies, load balancers and CDNs (content delivery networks) have their own buffering and compression settings. Run this check through every hop of the real path before launch. The -N flag stops curl's own buffering.

# Watch events arrive one by one (-N turns off curl's own buffering)
curl -N http://127.0.0.1:8001/stream

# Ask to resume after piece 3, the way a reconnecting browser does
curl -N -H "Last-Event-ID: m1:3" http://127.0.0.1:8001/stream

Quiet streams get cut

nginx closes a proxied connection when the server behind it sends nothing for proxy_read_timeout, 60 seconds by default. A model waiting in a busy queue can go quiet that long.

We set that timeout to 2 seconds and made the server go quiet for 3. Without comment lines, the stream died after the first event. With a comment line every second, the done event arrived at 3.0 seconds.

The HTML standard suggests a comment line every 15 seconds or so, because some proxies drop idle connections. Pick any interval well below the shortest idle timeout on your path.

The browser reconnects after the reply ends

The standard tells EventSource to reconnect whenever the response ends, even after a normal finish. So when your server closes a finished stream, the browser asks again. A careless server then streams the whole reply a second time.

Two fixes work together. The app calls close() on the EventSource when the done event arrives. The server answers 204 No Content when the last id says the reply is finished, which tells the browser to stop.

Here is the chat service's event writer with that resume logic. JSON-encoding the text keeps each data field on one line, even when the reply contains a line break.

import json

def sse_event(event_id, kind, data):
    body = json.dumps(data, ensure_ascii=False)  # always one line
    return f"id: {event_id}\nevent: {kind}\ndata: {body}\n\n"

def stream_reply(message_id, pieces, after=0):
    seq = 0
    for text in pieces:
        seq += 1
        if seq <= after:  # the app already has this piece
            continue
        yield sse_event(f"{message_id}:{seq}", "token", {"text": text})
    yield sse_event(f"{message_id}:end", "done", {"pieces": seq})

def resume_after(message_id, last_event_id):
    # "m1:2" means the app has pieces 1 and 2 of message m1
    if not last_event_id or not last_event_id.startswith(message_id + ":"):
        return 0  # fresh start
    tail = last_event_id[len(message_id) + 1:]
    return int(tail) if tail.isdigit() else None  # None: reply already done

pieces = ["Namaste", "! Day 1:", " Madikeri\nDay 2: Abbey Falls"]
after = resume_after("m1", "m1:1")
print("resume after:", after)
for event in stream_reply("m1", pieces, after=after):
    print(event, end="")
print("after a done id:", resume_after("m1", "m1:end"))
print("id from another message:", resume_after("m1", "m7:2"))

Running it on Python 3.13 prints pieces 2 and 3 and the done event, skipping piece 1. The line break inside piece 3 stays escaped inside one data line. The done id gives None, the signal to answer 204, and an id from another message gives 0, a fresh start.

In our test server, a reconnect with Last-Event-ID m1:3 received only pieces 4, 5 and done. A reconnect with m1:end got status 204 and no body.

Resuming works only inside one EventSource

The last event id lives inside each EventSource object and starts empty. If the page reloads or the app creates a new EventSource, no Last-Event-ID header is sent. Pass the last id yourself, for example in the stream URL.

MDN, the Mozilla web docs, notes that without HTTP/2 browsers allow only 6 SSE connections per domain across all tabs, as of October 2026. Serve streams over HTTP/2, the newer HTTP version that carries many requests over one connection. There the server sets the limit, and the HTTP/2 standard, RFC 9113, recommends allowing at least 100.

Stopping needs care too. In a simple blocking server like our Python test, the server learns the app has gone only when a write fails with BrokenPipeError. Async frameworks can report a disconnect sooner. Either way, cancel the model call, or you keep paying for unread tokens.

Walk backwards from the newest turn and keep turns until the next one would not fit. Recent messages always win, and the kept turns form one unbroken run.

In the code below, count_tokens counts words, a stand-in so the example runs anywhere. In production, use your model's tokenizer, because tokens and words differ.

def count_tokens(text):
    # Stand-in for testing: counts words. Use your model's tokenizer in production.
    return len(text.split())

def build_context(system, summary, turns, new_msg, limit=16_000, reply_room=1_000):
    budget = limit - reply_room - count_tokens(system) - count_tokens(new_msg)
    if summary:
        budget -= count_tokens(summary)
    kept = []
    for turn in reversed(turns):  # newest turn first
        cost = count_tokens(turn)
        if cost > budget:
            break
        kept.append(turn)
        budget -= cost
    kept.reverse()
    parts = [system, summary, *kept, new_msg]
    return [p for p in parts if p], len(turns) - len(kept)

turns = ["word " * 40, "word " * 30, "word " * 20, "word " * 10]   # oldest first
parts, dropped = build_context("Be brief.", "Wedding in Goa, 14 Feb, 80 guests.", turns,
                               "Find a caterer.", limit=100, reply_room=20)
print("parts sent:", len(parts), "turns dropped:", dropped)
print("tokens sent:", sum(count_tokens(p) for p in parts), "of", 100 - 20, "allowed")

Running it prints parts sent: 6 and turns dropped: 1, then tokens sent: 72 of 80 allowed. The second value returned is the count of older turns left out.

What just happened? The budget started at 68 after the system prompt, summary and new message. The three newest turns used 60, leaving 8. The 40-word oldest turn did not fit, so the loop stopped there.

A model gateway is one internal service that every model call passes through. It picks the model, holds the keys, applies timeouts, switches to a backup when a model fails, and records tokens and cost.

Without a gateway, keys spread across services, costs are hard to add up, and switching models means changing code everywhere. With one, the team changes a single place. These are its jobs.

• Routing: short, simple messages go to a small, cheap model and harder ones go to a large model. A classifier, a small model that sorts text into labels, makes the choice.
• Timeouts: if no first token arrives within 3 seconds, a made-up target, give up on that call.
• Fallback: after a timeout or an error, try a second model or another region.
• Accounting: record tokens in, tokens out and cost for every call, tagged with the user and the feature.
• Keys and capacity: hold the model service keys in one place, and respect each model's own limits.

Retries need care with streaming. Once the user has seen some words, a silent retry would restart the reply from a different place. So the gateway retries only before the first token. After that, a failure is shown to the user with a retry button.

The gateway calls the model servers but does not run them. How those servers batch requests to go fast is covered in our guide on LLM inference system design.

Safety checks go in two places: on the user's message before the model sees it, and on the reply while it streams, before each part reaches the screen.

The input check should be fast and cheap. It refuses oversized messages and runs a small classifier for clearly harmful requests.

Checking a streamed reply is harder, because words leave as soon as they are written. This is the radio station's broadcast delay from earlier. The station airs everything a few seconds late, so a producer can cut a word before listeners hear it.

The chat service holds back the newest sentence, checks it, and only then sends it on. At 50 tokens a second (made up), holding back a 20-token sentence adds about 0.4 seconds. The first words wait for the first sentence too, so the 1.5-second target must include that wait.

If a check fails midway, the service stops the model call and swaps the shown text for a short, safe message. The flagged reply goes to a review queue for the trust and safety team.

Finally, treat every reply as untrusted text. OWASP's 2025 entry on improper output handling says to treat the model like any other user and to encode its output for where it is shown. The app renders replies as text, never as raw HTML.

You keep it running by using two or more regions, keeping chat servers stateless, and planning a response to every failure: fall back, resume, or tell the user plainly.

The team's two cloud regions are Mumbai and Hyderabad. Stateless means a chat server keeps nothing it cannot lose. Chats live in the database and stream buffers in a shared cache, so any server can pick up a reconnecting user.

Failover means moving traffic to a backup region when the main one fails. In the diagram, look at which arrows cross between the regions.

The Hyderabad chat store is a replica, a second copy of the database. The copying is asynchronous, which means the replica runs slightly behind. If Mumbai fails suddenly, the newest messages may be missing, so measure that lag and agree how much loss is acceptable.

Both regions here are in India. If you add one abroad, check data protection laws and customer contracts first. Here is the team's plan for each failure.

• Model slow or down: the gateway falls back to another model or region before the first token.
• Model fails mid-reply: save the partial reply as failed, and show it with a retry button.
• Connection drops: the app reconnects with the last event id, and the server replays from the cache.
• Region down: the router sends new traffic to the other region. The replay cache stayed in Mumbai, so Hyderabad marks any reply still streaming after a timeout as failed, with a retry button.
• Overload everywhere: move free users to the small model with shorter replies, so paid users keep full service.

That last plan is an example of what OWASP's unbounded consumption entry calls graceful degradation: keeping partial service under heavy load instead of failing completely.

At the made-up prices below, model calls for this app cost about Rs 6.6 lakh a day after three savings, or about Rs 4 per user a month. We count only model calls, because they grow with every token of every message. Price servers and storage separately.

The prices here are made up. The large model costs Rs 20 per million input tokens and Rs 60 per million output tokens. The small model costs a fifth of that, cached input costs a tenth, and every reply is 300 tokens.

Row 2 assumes a typical request carries the 1,000-token system prompt, a short summary, the latest few turns and the new message. That comes to about 2,500 tokens (made up).

Check any row yourself. Cost per message is input tokens times the input price, plus 300 times the output price, divided by one million. For the first row that is 6,000 times 20 plus 300 times 60, which gives Rs 0.138.

The last row uses prompt caching. The serving system keeps the work it did on a prompt start it has seen recently, and reuses it when a new request begins the same way. Keep the system prompt identical and first, or the cache misses.

vLLM, an open-source serving engine, calls this automatic prefix caching. Its docs say it speeds up reading the prompt, not writing the reply. If your provider bills cached input at a lower price, as the table assumes, check the exact discount on its price page.

Rs 6.6 lakh a day across 50 lakh daily users is about Rs 0.13 per user per day, or about Rs 4 a month. That is under the Rs 6 target, with room for growth.

Also cap reply length by plan, and watch cost per message daily. One prompt change can quietly double it.

Follow the same order as this guide: requirements, numbers and the main path, then each hard part with its trade-off.

1. 1

   Clarify the requirements

   Ask about users, languages, plans, history length, safety rules and the reply-time goal. Write both requirement lists down.

2. 2

   Estimate the load

   Peak messages per second, open streams, tokens per day and storage per year. Round hard.

3. 3

   Draw the main path

   App, load balancer, chat service, conversation store, model gateway and models. Walk one message through.

4. 4

   Explain streaming

   SSE over HTTP, event ids for resuming, comment lines and no proxy buffering, close() on done, and cancelling the model call on stop.

5. 5

   Control context and use

   A token budget with a running summary, plus request rate limits, open-stream limits and daily token quotas.

6. 6

   Add safety and failure plans

   Input checks, a hold-back buffer, fallback before the first token, and a tested second region.

7. 7

   Close with cost

   Cost per message, the biggest levers, and the metric you would watch daily.