Design LLM batch inference on a GPU cluster
PremiumDistributed infrastructureHard
- Continuous batching
- KV cache
- GPU scheduling
Quick answer
This distributed infrastructure interview question asks you to design LLM batch inference on a GPU cluster. Aim for a design that keeps working across many machines and failures, then explain partitioning, replication, coordination and trade-offs. Interviewers want to see how you reason about failures, consistency and scale.
The problem
Design LLM batch inference on a GPU cluster: one shared fleet serves a large language model to interactive callers who stream tokens and to batch callers who want results within 24 hours. The hard part is packing requests of very different lengths onto GPUs whose KV-cache memory runs out long before compute does.
A company serves one large language model to developers through a public API, and all of it runs on a shared GPU cluster. Interactive callers (chat apps, coding assistants, agents) send a request and wait for the answer, usually streamed token by token. Batch callers upload files of up to 100,000 requests (evaluations, document classification, data labelling) and only need the results within 24 hours, in exchange for a lower price.
The system must pack both kinds of requests onto GPUs to get the most tokens per GPU-hour and still meet latency targets for interactive users. It exposes a synchronous API (the caller waits on one HTTP request) and an asynchronous batch API (submit, poll, download). Request lengths vary from a few tokens to 16,000, output is generated one token at a time, and the GPU memory that holds each in-flight sequence's attention state (the KV cache) is the scarce resource. This question has been reported at Anthropic in candidate system design interviews.
What must the system do?
- Sync API: A caller sends one request (messages, max_tokens, stream flag) and receives the full completion, or a token stream over Server-Sent Events, on the same HTTP connection.
- Batch API: A caller submits up to 100,000 requests in one job, each with its own custom_id, gets a batch id back, polls status, and downloads a results file.
How well must it do it?
- Time to first token: p50 at most 500 ms and p99 at most 2 s for prompts up to 4,000 tokens, measured at the API edge.
- Time per output token: p50 at most 40 ms and p99 at most 60 ms for streaming requests, even while batch jobs run.
What scale should you design for?
Use these numbers in your estimates.
The scale numbers unlock with this questionUnlock
Key concepts
- Continuous batching
- KV cache
- GPU scheduling
- Chunked prefill
- Priority queues
- Autoscaling
How you’ll practise
- Requirements
- Estimate
- Data model and API
- Partitioning and replication
- Consistency and failures
- Wrap-up
Lala, personalized to you, gives a hint when you ask and reviews your design when you are ready.
What do interviewers look for?
In any distributed infrastructure design round.
- States the consistency and availability guarantees up front
- Explains partitioning and replication choices
- Describes how nodes agree and recover after failures
- Handles network partitions and slow nodes
- Plans for adding and removing capacity safely
- Names the trade-offs instead of hiding them
Questions people ask about this interview question
- Which design type is this question?
- It is a distributed infrastructure question. You design LLM batch inference on a GPU cluster and explain your choices the way you would in an interview.
- Is this question asked at entry level?
- It is marked Hard. It suits engineers preparing for senior interviews, where you are expected to drive the discussion and go deep on trade-offs.
- What should I cover first?
- Start with the guarantees the system must give and the failures it must survive. Agree on scope with the interviewer before you design anything.
- Can I practise it for free?
- It is a premium question. Sign up and unlock it once for 10 credits; it then stays unlocked. When you finish, Lala, your personalized AI mentor, can review your design for 30 credits.
Related questions
Updated · Written by the SevyDevy team