Design a 200TB media transfer pipeline
PremiumData pipelineHard
- Bulk data transfer
- Bandwidth math
- Egress cost
Quick answer
This data pipeline interview question asks you to design a 200TB media transfer pipeline. Aim for a design that moves data from sources to consumers reliably and on time, then explain ingestion, processing, storage and trade-offs. Interviewers want to see how you handle volume, late or bad data and failures along the way.
The problem
Design a 200TB media transfer pipeline that moves 200 TB of images and videos from a cloud object store to a rented GPU cluster in another data center, then preprocesses them into training shards. The hard parts are doing the bandwidth and cost math before choosing a method, never losing or corrupting a file, and keeping the GPUs fed once training starts.
The data belongs to the team and is already cleared for training. After the first bulk move, about 1 TB of new media arrives every day and must reach the training dataset too, and media deleted at the source must leave the dataset.
What must a 200TB media transfer pipeline do?
- Bulk transfer: Copy all 200 TB from the source bucket to staging storage next to the GPU cluster, resumable after any failure.
- Integrity: Prove every object arrived complete and unchanged, with a manifest of keys, sizes and checksums.
How well must it do it?
- Deadline: Training starts within 14 days, including transfer and preprocessing.
- Cost: Spend on transfer and staging is estimated up front and kept under $20,000 for the first month.
What scale should you design for?
Use these numbers in your estimates.
The scale numbers unlock with this questionUnlock
Key concepts
- Bulk data transfer
- Bandwidth math
- Egress cost
- Manifest and checksums
- Staging storage
- Training data shards
How you’ll practise
- Requirements
- Data volume
- Ingestion
- Storage and processing
- Serving and freshness
- Wrap-up
Lala, personalized to you, gives a hint when you ask and reviews your design when you are ready.
What do interviewers look for?
In any data pipeline design round.
- Clarifies sources, consumers and freshness needs
- Chooses batch, streaming or both with reasons
- Handles late, duplicate and malformed records
- Makes each stage safe to retry
- Explains storage formats and partitioning
- Plans monitoring, backfills and data quality checks
Questions people ask about this interview question
- Which design type is this question?
- It is a data pipeline question. You design a 200TB media transfer pipeline and explain your choices the way you would in an interview.
- Is this question asked at entry level?
- It is marked Hard. It suits engineers preparing for senior interviews, where you are expected to drive the discussion and go deep on trade-offs.
- What should I cover first?
- Start with the data sources, the consumers and how fresh the data must be. Agree on scope with the interviewer before you design anything.
- Can I practise it for free?
- It is a premium question. Sign up and unlock it once for 10 credits; it then stays unlocked. When you finish, Lala, your personalized AI mentor, can review your design for 30 credits.
Related questions
Updated · Written by the SevyDevy team