Design a distributed LLM training system
PremiumDistributed infrastructureHard
- 3D parallelism
- ZeRO sharding
- Async checkpointing
Quick answer
This distributed infrastructure interview question asks you to design a distributed LLM training system. Aim for a design that keeps working across many machines and failures, then explain partitioning, replication, coordination and trade-offs. Interviewers want to see how you reason about failures, consistency and scale.
The problem
Design a distributed LLM training system that pretrains a 70B-parameter model on 15 trillion tokens across about 4,096 GPUs within 60 days and $11 million. The hard parts are splitting state that fits on no single GPU, keeping one synchronous job alive through failures every few hours, and wasting no GPU time.
A research team wants to pretrain the model from scratch, and you own the training system. That means the pipeline that turns a raw web-and-code crawl into training tokens, the way the model and its optimizer state are split across a cluster of a few thousand GPUs, and the machinery that keeps one synchronous job running for weeks while hardware fails underneath it. This question was reported at Robinhood and OpenAI in candidate interviews for senior ML infrastructure roles.
Every constraint pulls on the others. The training state does not fit on any single GPU. A synchronous job only moves as fast as its slowest GPU. At this scale some piece of hardware breaks every few hours, and every restart of thousands of processes burns money. The budget is fixed in dollars and the deadline in days, so wasted GPU time is the real enemy.
What must a distributed LLM training system do?
- Distributed training: Run one synchronous pretraining job across the cluster that consumes 15 trillion tokens at the fixed global batch size.
- Checkpoint and resume: Save the full training state periodically and resume from any retained checkpoint, including on a different set of nodes.
How well must it do it?
- Efficiency: Sustain at least 40% model FLOPs utilisation (MFU) across the training GPUs.
- Deadline and cost: Finish within 60 days and $11 million of GPU spend, including spares, restarts and lost work.
What scale should you design for?
Use these numbers in your estimates.
The scale numbers unlock with this questionUnlock
Key concepts
- 3D parallelism
- ZeRO sharding
- Async checkpointing
- Fault tolerance
- MinHash dedup
- Straggler detection
How you’ll practise
- Requirements
- Estimate
- Data model and API
- Partitioning and replication
- Consistency and failures
- Wrap-up
Lala, personalized to you, gives a hint when you ask and reviews your design when you are ready.
What do interviewers look for?
In any distributed infrastructure design round.
- States the consistency and availability guarantees up front
- Explains partitioning and replication choices
- Describes how nodes agree and recover after failures
- Handles network partitions and slow nodes
- Plans for adding and removing capacity safely
- Names the trade-offs instead of hiding them
Questions people ask about this interview question
- Which design type is this question?
- It is a distributed infrastructure question. You design a distributed LLM training system and explain your choices the way you would in an interview.
- Is this question asked at entry level?
- It is marked Hard. It suits engineers preparing for senior interviews, where you are expected to drive the discussion and go deep on trade-offs.
- What should I cover first?
- Start with the guarantees the system must give and the failures it must survive. Agree on scope with the interviewer before you design anything.
- Can I practise it for free?
- It is a premium question. Sign up and unlock it once for 10 credits; it then stays unlocked. When you finish, Lala, your personalized AI mentor, can review your design for 30 credits.
Related questions
Updated · Written by the SevyDevy team