Design a credit-based GPU scheduling system

Design a credit-based GPU scheduling system for a shared 4,096-GPU cluster where 60 teams spend monthly credit budgets next to production inference. The hard parts are placing big all-or-nothing jobs, preempting with little lost work, and charging every GPU-minute exactly once.

Dozens of research and product teams share the cluster. Each team receives a monthly budget of credits, where one credit buys one GPU-hour at the standard training price. Teams submit jobs that range from a single-GPU notebook to a 2,048-GPU pre-training run, and production inference services draw GPUs from the same pool. The system decides which jobs run, where they run, which jobs are preempted when more important work arrives, and how every GPU-minute is charged against a budget.

The hard parts are the conflicts. Large training jobs need hundreds of GPUs at the same moment on well-connected hardware, while a steady stream of small jobs keeps fragmenting free capacity. Teams want guaranteed capacity, yet guaranteed capacity that sits idle wastes the most expensive hardware in the company. Preempting a job throws away the work done since its last checkpoint. The credit ledger must stay exact even when node agents crash, retry or report late. This question has been reported at Anthropic in candidate interviews for infrastructure roles.