Design a web crawler
PremiumBackend systemHard
- Web crawling
- Data collection
- Scalability
Quick answer
This backend system interview question asks you to design a web crawler. Aim for a design that meets its requirements at the stated scale, then explain its main components, data flow and trade-offs. Interviewers want to see how you scope the problem, size it and defend each choice.
The problem
Design a distributed web crawler that can crawl and index web pages at large scale while being polite, fault tolerant, and storage efficient.
What must a web crawler do?
- The crawler can start from seed URLs and discover new URLs from downloaded pages.
- The system should crawl HTML pages only.
5 more requirementsUnlock to see all
How well must it do it?
- The crawler should scale horizontally across many workers.
- The system should handle high crawl throughput reliably.
4 more requirementsUnlock to see all
What scale should you design for?
Use these numbers in your estimates.
The scale numbers unlock with this questionUnlock
Key concepts
- Web crawling
- Data collection
- Scalability
- URL frontier
- Duplicate detection
- Distributed systems
How you’ll practise
6 short steps · save and come back any time
- Requirements
- Estimate
- API and data
- High-level design
- Deep dives
- Wrap-up
Lala, personalized to you, gives a hint when you ask and reviews your design when you are ready.
What do interviewers look for?
In any backend system design round.
- Clarifies the requirements before drawing anything
- Uses a few numbers that actually change the design
- Draws a clear path from the client to storage
- Picks storage and caching with reasons, not by habit
- Explains what breaks first under load and how to fix it
- States trade-offs openly instead of claiming a perfect design
Questions people ask about this interview question
- Which design type is this question?
- It is a backend system question. You design a web crawler and explain your choices the way you would in an interview.
- Is this question asked at entry level?
- It is marked Hard. It suits engineers preparing for senior interviews, where you are expected to drive the discussion and go deep on trade-offs.
- What should I cover first?
- Start with the requirements: what the system must do, how well it must do it and which numbers change the design. Agree on scope with the interviewer before you design anything.
- Can I practise it for free?
- It is a premium question. Sign up and unlock it once for 10 credits; it then stays unlocked. When you finish, Lala, your personalized AI mentor, can review your design for 30 credits.
Related questions
Updated · Written by the SevyDevy team