Design a distributed file system like HDFS
PremiumDistributed infrastructureMedium
- HDFS
- NameNode
- Block replication
Quick answer
This distributed infrastructure interview question asks you to design a distributed file system like HDFS. Aim for a design that keeps working across many machines and failures, then explain partitioning, replication, coordination and trade-offs. Interviewers want to see how you reason about failures, consistency and scale.
The problem
Design a distributed file system like HDFS for a batch analytics cluster. It stores 10 PB of large files on hundreds of commodity servers and streams them to batch jobs. The hard parts are keeping all metadata on one coordinator without making it a bottleneck or single point of failure, and surviving constant disk and server failures without data loss.
Files are written once and then read many times. Low-latency random writes are not needed.
What must a distributed file system like HDFS do?
- File operations: create, append, open and read at any offset, list, rename and delete, in a hierarchical namespace of directories and files.
- Large sequential I/O: Files of hundreds of MB to many GB are written and read in sequence at full disk speed.
How well must it do it?
- Durability: No data loss when any single server or a whole rack fails.
- Metadata availability: Metadata failover within 1 minute when the active metadata server dies.
What scale should you design for?
Use these numbers in your estimates.
The scale numbers unlock with this questionUnlock
Key concepts
- HDFS
- NameNode
- Block replication
- Rack awareness
- Write pipeline
- Small files problem
How you’ll practise
- Requirements
- Estimate
- Data model and API
- Partitioning and replication
- Consistency and failures
- Wrap-up
Lala, personalized to you, gives a hint when you ask and reviews your design when you are ready.
What do interviewers look for?
In any distributed infrastructure design round.
- States the consistency and availability guarantees up front
- Explains partitioning and replication choices
- Describes how nodes agree and recover after failures
- Handles network partitions and slow nodes
- Plans for adding and removing capacity safely
- Names the trade-offs instead of hiding them
Questions people ask about this interview question
- Which design type is this question?
- It is a distributed infrastructure question. You design a distributed file system like HDFS and explain your choices the way you would in an interview.
- Is this question asked at entry level?
- It is marked Medium. Entry-level engineers can practise it, but it expects you to reason about requirements and trade-offs, not only list components.
- What should I cover first?
- Start with the guarantees the system must give and the failures it must survive. Agree on scope with the interviewer before you design anything.
- Can I practise it for free?
- It is a premium question. Sign up and unlock it once for 10 credits; it then stays unlocked. When you finish, Lala, your personalized AI mentor, can review your design for 30 credits.
Related questions
Updated · Written by the SevyDevy team