Design GPU fleet health monitoring and repair
PremiumCloud infrastructureHard
- GPU health probes
- Auto remediation
- Blast radius
Quick answer
This cloud infrastructure interview question asks you to design GPU fleet health monitoring and repair. Aim for a design that runs reliably and securely in the cloud at a sensible cost, then explain the services, regions, deployment and trade-offs. Interviewers want to see how you balance reliability, security and cost.
The problem
Design GPU fleet health monitoring and repair for a cloud that runs 100,000 GPU hosts for AI training and inference. It probes every GPU, collects telemetry, spots failing hardware and repairs it automatically. The hard part is repairing fast without one bad signal draining a large part of the fleet.
Each host has 8 GPUs joined by a fast GPU interconnect. Faults range from a GPU that drops off the PCIe bus, to uncorrectable memory (ECC) errors, overheating, slow interconnect links and silent slowdowns. Repairs escalate: reset the GPU, reboot the host, reimage it, then hand it to a data center technician.
What must the system do?
- Detect faults: Flag a hard GPU or host fault within 2 minutes, using passive counters, active probes and job error reports.
- Repair automatically: Run an escalating workflow (reset, reboot, reimage, technician) and return a host to service only after it passes a burn-in test.
How well must it do it?
- Bounded blast radius: No probe bug, bad rule or network split can let automation take more than 1% of a region's hosts out of service.
- Idempotent repair: A crashed or retried workflow never repeats a reboot for the same incident, and a host never has two workflows at once.
What scale should you design for?
Use these numbers in your estimates.
The scale numbers unlock with this questionUnlock
Key concepts
- GPU health probes
- Auto remediation
- Blast radius
- Idempotent workflows
- Circuit breaker
- Fleet telemetry
How you’ll practise
- Requirements
- Environments and topology
- Deployment pipeline
- Reliability and disaster recovery
- Observability and cost
- Wrap-up
Lala, personalized to you, gives a hint when you ask and reviews your design when you are ready.
What do interviewers look for?
In any cloud infrastructure design round.
- Clarifies availability, security and cost targets
- Chooses managed services with reasons
- Plans regions, zones and failover
- Covers identity, secrets and network boundaries
- Explains deployment, rollback and observability
- Keeps an eye on cost as usage grows
Questions people ask about this interview question
- Which design type is this question?
- It is a cloud infrastructure question. You design GPU fleet health monitoring and repair and explain your choices the way you would in an interview.
- Is this question asked at entry level?
- It is marked Hard. It suits engineers preparing for senior interviews, where you are expected to drive the discussion and go deep on trade-offs.
- What should I cover first?
- Start with the workloads, the availability target and the security and cost limits. Agree on scope with the interviewer before you design anything.
- Can I practise it for free?
- It is a premium question. Sign up and unlock it once for 10 credits; it then stays unlocked. When you finish, Lala, your personalized AI mentor, can review your design for 30 credits.
Related questions
- Design an application health check systemCloud infrastructure · EasyPremium
- Design a distributed software release orchestration platformCloud infrastructure · HardPremium
- Design a multi-cloud disaster recovery orchestration platformCloud infrastructure · HardPremium
- Design a global infrastructure change approval & risk assessment platformCloud infrastructure · HardPremium
Updated · Written by the SevyDevy team