Design a search indexing pipeline
PremiumData pipelineMedium
- Search indexing
- Inverted index
- Change data capture
Quick answer
This data pipeline interview question asks you to design a search indexing pipeline. Aim for a design that moves data from sources to consumers reliably and on time, then explain ingestion, processing, storage and trade-offs. Interviewers want to see how you handle volume, late or bad data and failures along the way.
The problem
Design a search indexing pipeline for a large online store: it builds search documents for 200 million products from the catalog, prices, stock and reviews, keeps price and stock fresh within a minute, and rebuilds the index with zero downtime. The hard parts are update order, joins across sources and full reindexing.
Shoppers who see an old price or an out-of-stock item lose trust, and a broken index means no sales. The search engine cluster (an Elasticsearch-style engine) is given; this question is about getting the right documents into it.
What must a search indexing pipeline do?
- Incremental updates: Apply product, price, stock and rating changes to the live index.
- Build documents: Combine catalog, price, stock and rating into one search document per product and country.
How well must it do it?
- Freshness: Price and stock changes are searchable within 60 seconds at p99; new products within 5 minutes.
- Ordering: An older update never overwrites a newer one in the index.
What scale should you design for?
Use these numbers in your estimates.
The scale numbers unlock with this questionUnlock
Key concepts
- Search indexing
- Inverted index
- Change data capture
- Elasticsearch
- Index aliases
- Denormalization
How you’ll practise
- Requirements
- Data volume
- Ingestion
- Storage and processing
- Serving and freshness
- Wrap-up
Lala, personalized to you, gives a hint when you ask and reviews your design when you are ready.
What do interviewers look for?
In any data pipeline design round.
- Clarifies sources, consumers and freshness needs
- Chooses batch, streaming or both with reasons
- Handles late, duplicate and malformed records
- Makes each stage safe to retry
- Explains storage formats and partitioning
- Plans monitoring, backfills and data quality checks
Questions people ask about this interview question
- Which design type is this question?
- It is a data pipeline question. You design a search indexing pipeline and explain your choices the way you would in an interview.
- Is this question asked at entry level?
- It is marked Medium. Entry-level engineers can practise it, but it expects you to reason about requirements and trade-offs, not only list components.
- What should I cover first?
- Start with the data sources, the consumers and how fresh the data must be. Agree on scope with the interviewer before you design anything.
- Can I practise it for free?
- It is a premium question. Sign up and unlock it once for 10 credits; it then stays unlocked. When you finish, Lala, your personalized AI mentor, can review your design for 30 credits.
Related questions
Updated · Written by the SevyDevy team