Design a kafka file watcher for millions of files
PremiumData pipelineHard
- Kafka
- Inotify
- Change detection
Quick answer
This data pipeline interview question asks you to design a kafka file watcher for millions of files. Aim for a design that moves data from sources to consumers reliably and on time, then explain ingestion, processing, storage and trade-offs. Interviewers want to see how you handle volume, late or bad data and failures along the way.
The problem
Design a file watcher that tracks changes across millions of files and publishes them as Kafka events. Agents detect creates, modifications and deletes on file servers and object storage, and consumers such as a search indexer and a virus scanner react to each change.
The hard parts are ordering per file, applying each change once downstream, and catching up after outages without missing a change.
What must a kafka file watcher for millions of files do?
- Detect every change: Capture create, modify, delete and rename on file servers and in buckets, and publish an event to Kafka for each change.
- Per-file ordering: Consumers see the events of one file in the order the changes happened.
How well must it do it?
- Effectively once: Duplicate events may reach Kafka, but each consumer applies each file version once.
- Freshness: 95% of changes are published within 30 seconds (60 seconds for buckets).
What scale should you design for?
Use these numbers in your estimates.
The scale numbers unlock with this questionUnlock
Key concepts
- Kafka
- Inotify
- Change detection
- Per key ordering
- Idempotent consumers
- Reconciliation scan
How you’ll practise
- Requirements
- Data volume
- Ingestion
- Storage and processing
- Serving and freshness
- Wrap-up
Lala, personalized to you, gives a hint when you ask and reviews your design when you are ready.
What do interviewers look for?
In any data pipeline design round.
- Clarifies sources, consumers and freshness needs
- Chooses batch, streaming or both with reasons
- Handles late, duplicate and malformed records
- Makes each stage safe to retry
- Explains storage formats and partitioning
- Plans monitoring, backfills and data quality checks
Questions people ask about this interview question
- Which design type is this question?
- It is a data pipeline question. You design a kafka file watcher for millions of files and explain your choices the way you would in an interview.
- Is this question asked at entry level?
- It is marked Hard. It suits engineers preparing for senior interviews, where you are expected to drive the discussion and go deep on trade-offs.
- What should I cover first?
- Start with the data sources, the consumers and how fresh the data must be. Agree on scope with the interviewer before you design anything.
- Can I practise it for free?
- It is a premium question. Sign up and unlock it once for 10 credits; it then stays unlocked. When you finish, Lala, your personalized AI mentor, can review your design for 30 credits.
Related questions
Updated · Written by the SevyDevy team