Design a batch job scheduler like airflow
PremiumData pipelineMedium
- DAG scheduling
- Workflow orchestration
- Task queues
Quick answer
This data pipeline interview question asks you to design a batch job scheduler like airflow. Aim for a design that moves data from sources to consumers reliably and on time, then explain ingestion, processing, storage and trade-offs. Interviewers want to see how you handle volume, late or bad data and failures along the way.
The problem
Design a batch job scheduler like Airflow: teams define pipelines as DAGs of tasks with schedules, and the scheduler starts each task when its upstream tasks finish, runs it on a worker pool and retries failures, for 5,000 DAGs. The hard parts are the midnight burst, tasks that run twice or never, and backfills.
The scheduler is shared by every data team. A stuck scheduler makes every dashboard in the company late, so it must survive the loss of any single machine.
What must a batch job scheduler like airflow do?
- DAGs: Teams define DAGs of tasks with dependencies, a schedule (cron or interval) and per-task retries and timeouts.
- Dispatch: A task starts when all its upstream tasks have succeeded, on a shared worker pool with per-team limits.
How well must it do it?
- Scheduling latency: A task starts within 10 seconds of becoming ready at p95, when worker capacity is free.
- Availability: No single machine failure stops scheduling; scheduler failover takes under 1 minute.
What scale should you design for?
Use these numbers in your estimates.
The scale numbers unlock with this questionUnlock
Key concepts
- DAG scheduling
- Workflow orchestration
- Task queues
- Heartbeats and leases
- Idempotent tasks
- Backfills
How you’ll practise
- Requirements
- Data volume
- Ingestion
- Storage and processing
- Serving and freshness
- Wrap-up
Lala, personalized to you, gives a hint when you ask and reviews your design when you are ready.
What do interviewers look for?
In any data pipeline design round.
- Clarifies sources, consumers and freshness needs
- Chooses batch, streaming or both with reasons
- Handles late, duplicate and malformed records
- Makes each stage safe to retry
- Explains storage formats and partitioning
- Plans monitoring, backfills and data quality checks
Questions people ask about this interview question
- Which design type is this question?
- It is a data pipeline question. You design a batch job scheduler like airflow and explain your choices the way you would in an interview.
- Is this question asked at entry level?
- It is marked Medium. Entry-level engineers can practise it, but it expects you to reason about requirements and trade-offs, not only list components.
- What should I cover first?
- Start with the data sources, the consumers and how fresh the data must be. Agree on scope with the interviewer before you design anything.
- Can I practise it for free?
- It is a premium question. Sign up and unlock it once for 10 credits; it then stays unlocked. When you finish, Lala, your personalized AI mentor, can review your design for 30 credits.
Related questions
Updated · Written by the SevyDevy team