Design a distributed rate limiter for an LLM API
PremiumDistributed infrastructureMedium
- Rate limiting
- Token bucket
- Token accounting
Quick answer
This distributed infrastructure interview question asks you to design a distributed rate limiter for an LLM API. Aim for a design that keeps working across many machines and failures, then explain partitioning, replication, coordination and trade-offs. Interviewers want to see how you reason about failures, consistency and scale.
The problem
Design a distributed rate limiter for an LLM API that enforces requests, input tokens, output tokens and concurrency per organization across 3 regions, in a few milliseconds. The hard part is that one request can cost 1,000 times another, and the output length is unknown until a stream ends.
You run the public API for a family of large language models. Hundreds of thousands of organizations call it from their own products: chat apps, coding agents and batch summarisers. Each organization is on a usage tier, and each tier limits requests per minute, input tokens per minute, output tokens per minute and concurrent requests. The limiter enforces all of these limits on every API call, in every region, before the request reaches a model server.
This question has been reported at Anthropic in candidate interviews. It looks like a classic rate limiter, but the cost of one request varies by more than 1,000 times (a 50-token prompt versus a 150,000-token document), and the output length is unknown until generation ends, which can take minutes for a long stream. The limiter must stay correct across hundreds of gateway nodes and three regions, and stop one customer's burst from hurting everyone else on the same tier.
What must a distributed rate limiter for an LLM API do?
- Multi-dimension limits: Enforce requests per minute (RPM), input tokens per minute (ITPM), output tokens per minute (OTPM) and concurrent in-flight requests per organization, all at once.
- Accurate charging: Charge input tokens from the actual prompt and output tokens that were actually generated, including streams that run for minutes or end early.
How well must it do it?
- Added latency: The rate-limit decision adds under 1 ms at p50 and under 5 ms at p99 to each request.
- Global scope: Limits apply per org across all 3 regions combined, not per region.
What scale should you design for?
Use these numbers in your estimates.
The scale numbers unlock with this questionUnlock
Key concepts
- Rate limiting
- Token bucket
- Token accounting
- Redis lua
- Concurrency limits
- Multi region
How you’ll practise
- Requirements
- Estimate
- Data model and API
- Partitioning and replication
- Consistency and failures
- Wrap-up
Lala, personalized to you, gives a hint when you ask and reviews your design when you are ready.
What do interviewers look for?
In any distributed infrastructure design round.
- States the consistency and availability guarantees up front
- Explains partitioning and replication choices
- Describes how nodes agree and recover after failures
- Handles network partitions and slow nodes
- Plans for adding and removing capacity safely
- Names the trade-offs instead of hiding them
Questions people ask about this interview question
- Which design type is this question?
- It is a distributed infrastructure question. You design a distributed rate limiter for an LLM API and explain your choices the way you would in an interview.
- Is this question asked at entry level?
- It is marked Medium. Entry-level engineers can practise it, but it expects you to reason about requirements and trade-offs, not only list components.
- What should I cover first?
- Start with the guarantees the system must give and the failures it must survive. Agree on scope with the interviewer before you design anything.
- Can I practise it for free?
- It is a premium question. Sign up and unlock it once for 10 credits; it then stays unlocked. When you finish, Lala, your personalized AI mentor, can review your design for 30 credits.
Related questions
Updated · Written by the SevyDevy team