Design a lab machine fleet monitor
PremiumDistributed infrastructureMedium
- Heartbeats
- Lease based work claiming
- Fencing tokens
Quick answer
This distributed infrastructure interview question asks you to design a lab machine fleet monitor. Aim for a design that keeps working across many machines and failures, then explain partitioning, replication, coordination and trade-offs. Interviewers want to see how you reason about failures, consistency and scale.
The problem
Design a lab machine fleet monitor for thousands of test machines that pull work, download and verify a payload, run it and report back, like the device labs hardware teams use for test runs. A dashboard shows each machine as downloading, running, crashed or hung, with logs and remote reboot. The hard part is telling dead from slow and never running one job twice at once.
Each machine runs a small agent that you design. Machines sit in 4 labs, and each lab must keep working if its link to the central region drops.
What must a lab machine fleet monitor do?
- Pull work: An idle agent asks for a job from its hardware pool, downloads the payload, verifies it, runs it and reports the result.
- Live states: The dashboard shows idle, downloading, verifying, running, crashed, hung and offline for every machine, filtered by lab and pool.
How well must it do it?
- No double runs: A job never has two machines running it at the same time, and a result from a machine that lost the job is rejected.
- Fast detection: A silent machine shows offline within 30 seconds, and a stalled job shows hung within its stall timeout.
What scale should you design for?
Use these numbers in your estimates.
The scale numbers unlock with this questionUnlock
Key concepts
- Heartbeats
- Lease based work claiming
- Fencing tokens
- SHA 256 verification
- Hung job detection
- Out of band power control
How you’ll practise
- Requirements
- Estimate
- Data model and API
- Partitioning and replication
- Consistency and failures
- Wrap-up
Lala, personalized to you, gives a hint when you ask and reviews your design when you are ready.
What do interviewers look for?
In any distributed infrastructure design round.
- States the consistency and availability guarantees up front
- Explains partitioning and replication choices
- Describes how nodes agree and recover after failures
- Handles network partitions and slow nodes
- Plans for adding and removing capacity safely
- Names the trade-offs instead of hiding them
Questions people ask about this interview question
- Which design type is this question?
- It is a distributed infrastructure question. You design a lab machine fleet monitor and explain your choices the way you would in an interview.
- Is this question asked at entry level?
- It is marked Medium. Entry-level engineers can practise it, but it expects you to reason about requirements and trade-offs, not only list components.
- What should I cover first?
- Start with the guarantees the system must give and the failures it must survive. Agree on scope with the interviewer before you design anything.
- Can I practise it for free?
- It is a premium question. Sign up and unlock it once for 10 credits; it then stays unlocked. When you finish, Lala, your personalized AI mentor, can review your design for 30 credits.
Related questions
Updated · Written by the SevyDevy team