Design ChatGPT
PremiumLLM systemHard
- Token streaming
- KV cache
- Prefix caching
Quick answer
This LLM system interview question asks you to design ChatGPT. Aim for a design that uses language models to serve users well, then explain retrieval, prompting, serving, evaluation and trade-offs. Interviewers want to see how you control quality, cost and latency, and how you evaluate the results.
The problem
Design ChatGPT: a conversational AI service, like ChatGPT or Claude, where 100 million daily users send messages and watch replies stream word by word. The hard parts are streams that survive flaky networks, history that outgrows a 128K context window, and GPU memory and cost.
Users open the web or mobile app, start or continue a conversation, type a message and see the answer begin within a fraction of a second. They come back days later on another device and expect every conversation to be there, and they expect very long conversations to keep working. This question has been reported at OpenAI and Anthropic in candidate interviews.
You own everything from the client connection to the GPU fleet that runs the models. The service has a free tier and a paid tier, two models (a large default model and a smaller, cheaper one) and runs in three regions. Every request holds a GPU-backed stream open for several seconds, GPU memory and GPU cost are the binding limits, and users on flaky mobile networks still expect an answer that never disappears halfway through.
What must the system do?
- Send and stream: A user sends a message in a conversation and receives the reply as a stream of text chunks that ends with a completion event.
- Resume after disconnect: If the connection drops or the page reloads mid-answer, the client reattaches and shows the full answer with no text lost or repeated.
How well must it do it?
- Latency: Time to first token p50 at most 500 ms and p99 at most 2 s for paid users at peak (free p99 at most 5 s); streams sustain at least 30 tokens/s at p99.
- Availability: Sending a message is available 99.9% per month; reading history 99.95%.
What scale should you design for?
Use these numbers in your estimates.
The scale numbers unlock with this questionUnlock
Key concepts
- Token streaming
- KV cache
- Prefix caching
- Context window
- Model routing
- SSE
How you’ll practise
- Problem framing
- Data and retrieval
- Prompting and orchestration
- Evaluation and safety
- Serving, cost and latency
- Wrap-up
Lala, personalized to you, gives a hint when you ask and reviews your design when you are ready.
What do interviewers look for?
In any LLM system design round.
- Defines the user task and what a good answer is
- Explains how relevant context reaches the model
- Plans evaluation before and after launch
- Controls cost and latency with caching and routing
- Covers safety, privacy and abuse
- Handles failures and fallbacks gracefully
Questions people ask about this interview question
- Which design type is this question?
- It is an LLM system question. You design ChatGPT and explain your choices the way you would in an interview.
- Is this question asked at entry level?
- It is marked Hard. It suits engineers preparing for senior interviews, where you are expected to drive the discussion and go deep on trade-offs.
- What should I cover first?
- Start with the user task, what a good answer looks like and how it will be measured. Agree on scope with the interviewer before you design anything.
- Can I practise it for free?
- It is a premium question. Sign up and unlock it once for 10 credits; it then stays unlocked. When you finish, Lala, your personalized AI mentor, can review your design for 30 credits.
Related questions
Updated · Written by the SevyDevy team