Design a data lake on object storage
PremiumData pipelineEasy
- Data lake
- Object storage
- Apache iceberg
Quick answer
This data pipeline interview question asks you to design a data lake on object storage. Aim for a design that moves data from sources to consumers reliably and on time, then explain ingestion, processing, storage and trade-offs. Interviewers want to see how you handle volume, late or bad data and failures along the way.
The problem
Design a data lake on object storage: one place where a company lands data from 30 sources, keeps it as open-format tables and lets analysts and batch jobs query it with SQL engines. The hard parts are table layout, safe writes on storage that has no transactions, small files and storage cost.
Today data sits in scattered database exports and spreadsheets, and each team copies what it needs. The lake replaces that with shared, documented tables that every engine can read.
What must a data lake on object storage do?
- Ingest: Land data from 12 database exports, 15 app event streams and 3 SaaS exports, keeping the original files unchanged.
- Tables: Expose cleaned and curated data as SQL tables that Spark and Trino-style engines can read and write.
How well must it do it?
- Consistent reads: A reader never sees half of a write; a job commits all of its files or none.
- Query speed: A typical query over the last 30 days of one table finishes in under 30 seconds.
What scale should you design for?
Use these numbers in your estimates.
The scale numbers unlock with this questionUnlock
Key concepts
- Data lake
- Object storage
- Apache iceberg
- Parquet
- Partitioning
- Small file compaction
How you’ll practise
- Requirements
- Data volume
- Ingestion
- Storage and processing
- Serving and freshness
- Wrap-up
Lala, personalized to you, gives a hint when you ask and reviews your design when you are ready.
What do interviewers look for?
In any data pipeline design round.
- Clarifies sources, consumers and freshness needs
- Chooses batch, streaming or both with reasons
- Handles late, duplicate and malformed records
- Makes each stage safe to retry
- Explains storage formats and partitioning
- Plans monitoring, backfills and data quality checks
Questions people ask about this interview question
- Which design type is this question?
- It is a data pipeline question. You design a data lake on object storage and explain your choices the way you would in an interview.
- Is this question asked at entry level?
- It is marked Easy, so it suits engineers who are new to data pipeline interviews. It is also a good warm-up before harder questions.
- What should I cover first?
- Start with the data sources, the consumers and how fresh the data must be. Agree on scope with the interviewer before you design anything.
- Can I practise it for free?
- It is a premium question. Sign up and unlock it once for 10 credits; it then stays unlocked. When you finish, Lala, your personalized AI mentor, can review your design for 30 credits.
Related questions
Updated · Written by the SevyDevy team