Design a Web Crawler: Politeness, Dedup and the URL Frontier

A simple crawler fetches as fast as it can, from whichever site its queue hands it next. Small servers slow down or crash, and their owners block it.

Web crawler system design is how you avoid that. You plan the crawler so it fetches millions of pages, stays gentle with every site and wastes nothing on repeats.

Picture a common setup. A small team builds a search engine for Indian-language pages in Odia, Hindi, Bengali and Telugu. The setup is made up, but the problems are real.

A crawler is a program that downloads web pages, finds the links inside them, and downloads those too. The team's first crawler was a weekend project: one loop around one queue.

It started from 500 seed URLs, the hand-picked starting pages, mostly Odia news and literature sites. Over one weekend, with 64 threads (parallel workers inside one program), it fetched 2 million pages. It also caused four problems (numbers made up):

• An Odia poetry site, run by one person on a cheap server, got 30 requests a second for six hours. It crashed twice, and its owner blocked the crawler.
• About 35% of the pages were repeats. One news story sat at its plain address, at a print version, and at copies with tracking tags added.
• One event calendar had a next month link on every page. The crawler followed it to the year 2400 and saved 90,000 empty pages.
• A big Hindi news site was barely crawled, because thousands of small blogs had filled the queue first.

The loop itself was not wrong. It was missing three things: a smarter to-do list, good manners, and a memory of what it had already seen. This guide builds all three.

Web crawler system design is planning a system that downloads a large part of the web, keeps it fresh, and stays gentle with every site it visits. It tests queues, hashing and rate control in one question.

Search engines are the classic user. Web archives, price trackers and research projects run crawlers too. Each has the same core loop: pick a URL, fetch it, pull out the links, and add the new ones.

That loop fits in ten lines of code. The hard part is everything around it at scale. Before drawing any boxes, the team writes down what its crawler actually needs.

• Scale: about 200 million pages in Indian languages, each refreshed roughly once a month.
• Politeness: never overload a site, and obey each site's robots.txt rules.
• Freshness: news homepages checked every hour or so, old archive pages far less often.
• No waste: skip pages already seen, whether by address or by content.
• Survive failures: a crashed machine must not lose the to-do list or restart the whole crawl.

Some things are out of scope. Building the search index, ranking results and running each page's JavaScript are separate systems. The crawler fetches HTML and hands it on to them.

A web crawler has six main parts: a URL frontier, fetchers, DNS and robots.txt caches, a page store, a parser, and two duplicate checks. Each part does one job and passes work along.

The textbook Introduction to Information Retrieval lists five modules: the frontier, DNS resolution, fetching, parsing and duplicate elimination. A housekeeping thread also checkpoints the crawl, saving its state to disk once every few hours.

Follow one URL around the diagram below. It leaves the frontier, gets fetched and parsed, and its new links return to the frontier only if they are unseen.

Two terms in the diagram need a word. DNS is the internet's phone book: it turns a hostname into a server's IP address. A fetcher is a worker that downloads pages over HTTP, the protocol browsers use to ask servers for pages.

A cache is a nearby copy you keep so you do not have to ask again. Looking up DNS for every single page would add delay, so fetchers reuse recent answers for each host.

The page store keeps raw HTML in object storage, which is cheap storage for large files. The parser reads from it later, so a slow parser never holds up the fetchers.

Start with pages per second, then storage, then the size of the seen-URL memory. These rough numbers tell you how many machines you need and which parts must be spread out.

A Bloom filter, in the last row, is a compact memory of seen URLs that is sometimes wrong in one safe direction. The section on duplicate checks in code explains it.

Seventy-seven pages a second does not sound like much. The catch is politeness. You cannot spread those fetches over the web however you like.

Pause and predict: the crawler waits 2 seconds between requests to one host. How long does one Hindi news site with 5 million pages take to crawl once?

One host gives at most 43,200 pages a day, because a day has 86,400 seconds. Five million pages then take about 116 days, nearly four months.

That one sum shapes the whole design. Big sites cannot be crawled in full every month, so the crawler must choose which of their pages matter most.

The URL frontier is the crawler's to-do list: every URL waiting to be fetched, ordered so important pages come first and no host is visited too often. It is two sets of queues joined by a timer.

Think of a Pulse Polio drive, where teams go house to house giving polio drops. Picture a supervisor with a list of lanes to cover, crowded lanes first. Teams mark a P on houses they have covered.

Map it part by part. The list of lanes is the frontier. Crowded lanes first is the priority order. The P mark is the seen-URL set.

Where the picture breaks: a house has one door, but one web page can live at many addresses, so a crawler also needs a content check. And a lane never needs rest between visits, while a website does.

Front queues decide what matters

Front queues handle priority. A prioritiser gives each new URL a score and drops it into one of several queues, from most to least urgent. The team uses three: hot, normal and cold.

The score comes from simple signals. The textbook bases it on fetch history, so pages that changed often rank higher. The team adds page type and the date in the site's sitemap.

A sitemap is an XML file, a text format built from tags, where a site lists its own URLs. The sitemaps protocol allows up to 50,000 URLs and 50 MB uncompressed per file. Each URL has an optional lastmod date for its last change.

Back queues decide when

Back queues handle politeness. Each back queue holds URLs from exactly one host. The textbook passes on a rule of thumb from the designers of the Mercator crawler: three back queues per crawler thread.

A heap sits on top of the back queues. A heap is a structure that always hands you its smallest item quickly. Here, each entry is a host plus the earliest time it may be visited again.

A fetcher thread takes the host at the top of the heap and waits until its time comes. It fetches one URL from that host's back queue, sets the host's next allowed time, and puts it back.

In the diagram, trace a URL from the left. Priority is settled in the front queues. Timing is settled at the heap, just before the fetch.

When a back queue runs empty, the router refills it. It pulls URLs from the front queues, favouring urgent ones. A URL whose host already has a back queue joins that queue.

The router keeps pulling until it meets a URL whose host has no back queue yet. That URL goes into the empty queue, which now belongs to its host.

A polite crawler obeys each site's robots.txt file and leaves a gap between requests to the same host. It also names itself honestly and slows down when a server shows strain.

Read robots.txt first

Robots.txt is a plain text file at the top level of a site that tells crawlers which paths they may fetch. Its rules became a standard in September 2022: RFC 9309, the Robots Exclusion Protocol. An RFC is a numbered document that defines an internet standard.

Rules are grouped by user agent, the name a crawler gives itself, such as IndicBot. A crawler with no group of its own must obey the group whose user agent is a star. These rules from the standard matter most:

• The most specific rule wins, meaning the matching rule with the longest path. If an allow rule and a disallow rule are equivalent, the standard says allow should win.
• You may cache the file, but should not use a cached copy for more than 24 hours, unless the file cannot be reached.
• If fetching the file returns a 4xx error, such as 404 not found, the crawler may fetch anything on the site.
• If it returns a 5xx server error, the crawler must assume it may fetch nothing. After a long outage, such as 30 days, it may treat the file as missing or keep a cached copy.
• Crawlers should follow at least five redirects to find the file, and must be able to read at least 500 KiB (about half a megabyte) of it.

Check robots.txt just before each fetch, not when a URL joins the queue. The textbook notes that a URL can wait in the frontier for days or even weeks, and the file may change meanwhile.

Robots.txt is not a lock, either. RFC 9309 says its rules are not a form of access authorization. Obeying them is the agreed manners of the web, and sites that see them ignored often block the crawler.

Name your crawler honestly. RFC 9309 says the crawler's name in robots.txt should appear inside the User-Agent header it sends. Site owners can then find it in their logs and write rules for it.

Leave a gap between requests

The robots.txt standard says nothing about request rate. The pace is your choice. The textbook suggests a gap an order of magnitude larger than your last fetch from that host, so about ten times as long.

The team's rule: wait the larger of 2 seconds or ten times the last fetch time. A host answering in 0.1 seconds gets one request every 2 seconds. A host taking 1 second gets one every 10.

This rule slows down by itself when a server struggles, because slow answers mean longer gaps. The team adds three more guards.

• An HTTP 429 reply means too many requests. RFC 6585 says it may carry a Retry-After header, a labelled line in the reply, saying how long to wait, so the crawler waits at least that long.
• Repeated 503 replies or timeouts double the host's gap, up to a ceiling, until normal answers return. Google documents the same habit: many 500, 503 or 429 replies make its crawler slow down for the whole hostname, as of 19 October 2026.
• Each host has a daily page budget, so one giant site cannot keep every fetcher busy.

A crawler avoids repeats with two checks: a seen-URL test before fetching, and a content fingerprint test after fetching. The first saves bandwidth, and the second catches one page living at many addresses.

The seen-URL set is the P mark on the door. But each URL must be cleaned first, or the same door ends up with several different marks.

Normalise URLs before checking

URL normalisation means rewriting a URL into one standard form. The team lowercases the scheme (http or https) and the hostname, and drops the part after the hash sign.

It also removes tracking tags and sorts what is left of the query, the part after the question mark. Without this, a story shared with a tracking tag and the same story without one look like two pages.

Fingerprint the content after fetching

A fingerprint is a short hash of the page body. A hash function such as SHA-256 turns any bytes into a fixed-length code. Two bodies with the same SHA-256 code are, in practice, the same bytes.

The textbook calls this step the content-seen test. It also catches copies at addresses that normalisation cannot link, such as a print version of a story.

Shingles, in the last row, are short overlapping runs of words. The textbook uses them for near-duplicates, pages that differ only in a date, an advert or a visitor counter. Pages sharing most shingles count as copies.

That completes the beginner path: a frontier for order and timing, robots.txt and gaps for manners, and two duplicate checks for memory. The sections below turn each one into tested code and numbers.

You keep a heap of hosts ordered by their next allowed visit time, and a waiting list of URLs per host. This sketch uses only Python's standard library and runs as shown.

import heapq, time
ready = []      # heap of (time the host may be visited again, host)
waiting = {}    # host -> URLs still to fetch from that host
next_ok = {}    # host -> earliest time of its next visit

def add_url(host, url):
    if host not in waiting:
        waiting[host] = []
        heapq.heappush(ready, (next_ok.get(host, time.monotonic()), host))
    waiting[host].append(url)

def next_url():
    ready_at, host = heapq.heappop(ready)      # the host that is ready soonest
    time.sleep(max(0.0, ready_at - time.monotonic()))
    return host, waiting[host].pop(0)

def after_fetch(host, seconds_taken, min_gap=2.0):
    gap = max(min_gap, 10 * seconds_taken)     # the team's politeness rule
    next_ok[host] = time.monotonic() + gap
    if waiting[host]:
        heapq.heappush(ready, (next_ok[host], host))
    else:
        del waiting[host]

start = time.monotonic()
for path in ["/a1", "/a2", "/a3"]:
    add_url("a.example", path)
add_url("b.example", "/b1")
while ready:
    host, path = next_url()
    print(f"{time.monotonic() - start:.1f}s {host}{path}")
    after_fetch(host, 0.05)                    # pretend each fetch took 50 ms

Run it with python3 and it prints four lines in about 4 seconds. Tested on Python 3.11.17, 3.12.3 and 3.13.16, each run printed exactly this:

0.0s a.example/a1
0.0s b.example/b1
2.0s a.example/a2
4.0s a.example/a3

What just happened? Host a.example had three URLs, so its second and third fetches waited 2 seconds each. Host b.example was ready at once, so it slipped into the gap at 0.0 seconds.

The next_ok table matters more than it looks. It remembers each host's next allowed time even after its queue empties. Without it, a host that gets a new link a moment later is fetched at once.

That is an easy mistake to make. In our test, a version without this table fetched a re-added host twice at 0.0 seconds. With next_ok, the second fetch waited the full 2 seconds.

The code uses time.monotonic, a clock that never jumps backwards, so a change to the system clock cannot break the waits. It is single-threaded. With many fetcher threads, guard the heap and the tables with a lock.

Python ships a robots.txt parser, urllib.robotparser, but its answers depend on your Python version. In Python 3.11 and 3.12, the first matching rule wins, not the most specific one.

This short check shows the difference. It parses rules that block an archive but allow its 2026 folder, then asks about two pages and the crawl delay.

from urllib.robotparser import RobotFileParser

rules = """
User-agent: *
Disallow: /archive/
Allow: /archive/2026/
Crawl-delay: 5
"""
rp = RobotFileParser()
rp.parse(rules.splitlines())
print(rp.can_fetch("IndicBot", "https://news.example/archive/2026/story"))
print(rp.can_fetch("IndicBot", "https://news.example/archive/2019/story"))
print(rp.crawl_delay("IndicBot"))

On Python 3.13.16 it prints True, False and 5. On Python 3.11.17 and 3.12.3 it prints False, False and 5, because the Disallow line comes first and wins.

Python's changelogs list RFC 9309 support in urllib.robotparser under releases 3.13.14 (10 June 2026) and 3.14.5 (10 May 2026). Earlier 3.13 and 3.14 releases still use the first match.

The library's documentation page links RFC 9309 but has no Changed in version note for this behaviour, as of 19 October 2026. Pin your Python version, and run this check in your build.

The status code of the robots.txt reply matters too. We served each code from a local test server and asked the parser about one page. Google's rules are from its robots.txt page, as of 19 October 2026.

Two practical points follow. A 429 is the server asking you to slow down, so treat it as a reason to wait, not as permission. And the parser never refreshes itself: build a new parser and call read() at least every 24 hours.

Python's documentation suggests its mtime() method for exactly this, to tell long-running spiders when the file was last fetched. Note that RFC 9309 never mentions the Crawl-delay line, though Python's crawl_delay() reads it.

Reusing one parser object for refreshes is a trap, undocumented as of 19 October 2026. We served one parser 200, then 503, then 200 again, calling read() each time.

On 3.11.17 and 3.12.3, the 503 left the old rules in force. On 3.13.16, the 503 blocked the site, and the next 200 did not unblock it. The changelogs list this 5xx block under 3.13.15 and 3.14.7 (5 August 2026).

A changed file misfires too. In a second test, we served one parser two different files, each with a 200. 3.11.17 and 3.12.3 kept the first group for all crawlers (User-agent: *), so a newly added Disallow line was ignored. 3.13.16 merged old and new rules, so a removed Disallow line still blocked. Create a fresh parser for each refresh.

Two Python sets are enough to show the logic. One holds normalised URLs, the other holds SHA-256 digests of page bodies. In production the sets become a Bloom filter or a shared key-value store.

import hashlib
from urllib.parse import urlsplit, urlunsplit, parse_qsl, urlencode
TRACKING = {"utm_source", "utm_medium", "utm_campaign", "fbclid"}
seen_urls, seen_bodies = set(), set()

def normalize(url):
    p = urlsplit(url)
    query = sorted((k, v) for k, v in parse_qsl(p.query) if k not in TRACKING)
    return urlunsplit((p.scheme.lower(), p.netloc.lower(), p.path or "/",
                       urlencode(query), ""))       # "" drops the #fragment

def url_is_new(url):                                # runs before the fetch
    key = normalize(url)
    new = key not in seen_urls
    seen_urls.add(key)
    return new

def body_is_new(body):                              # runs after the fetch
    digest = hashlib.sha256(body).hexdigest()
    new = digest not in seen_bodies
    seen_bodies.add(digest)
    return new

for u in ["https://News.example/story?id=7&utm_source=x",
          "https://news.example/story?utm_medium=y&id=7#top",
          "https://news.example/story?id=8"]:
    print(url_is_new(u), normalize(u))
print(body_is_new(b"<p>Same story</p>"), body_is_new(b"<p>Same story</p>"))

Run it and you should see the first two URLs collapse to one key. Tested on Python 3.11.17, 3.12.3 and 3.13.16, the output was:

True https://news.example/story?id=7
False https://news.example/story?id=7
True https://news.example/story?id=8
True False

What just happened? The capital N, the tracking tags, the parameter order and the #top fragment all vanished, so the second URL was seen. The same body bytes were new once, then seen.

It is a sketch, and our edge tests found three limits. It keeps default ports such as :443, drops empty parameters like a=, and reorders repeated keys. An app that reads repeated keys in order would see two different pages merged, so keep a per-host exception list.

Bloom filters for a billion URLs

A Bloom filter is a bit array plus a few hash functions. Adding a URL sets a few bits. Asking about a URL answers either maybe seen or definitely new.

At a 1% false positive rate it needs about 9.6 bits per URL, so a billion URLs fit in about 1.2 GB. Check the bits per URL yourself; this command prints 9.59.

python3 -c "import math; print(round(-math.log(0.01) / math.log(2)**2, 2))"

The team lets sitemap URLs bypass the filter, so a false positive cannot hide a page the site itself listed. The exact 8-byte hash set is the other option, at about 8 GB for a billion URLs.

Split the work by hostname: hash each host to one crawler machine, and that machine owns everything about it. The host's back queue, timer, robots.txt copy and DNS answers all live in one place.

This matters because politeness needs one clock per host. If two machines both fetched from the poetry site, each would think the site was resting while the other knocked. One owner per hostname rules that out.

Hostnames are the right unit for robots.txt too. Google's documentation says a robots.txt file applies only to the host, protocol and port where it is posted. Two hostnames can still share one server, so consider a cap per IP address as well.

The textbook calls the routing part a host splitter. After the URL filter, it sends each new URL to the machine responsible for its host.

Content fingerprints cannot be split by host, because the same article can sit on different sites. The textbook splits them by a property of the fingerprint itself, such as the remainder after dividing its value by the number of machines.

Most content checks then become remote calls to another machine. The textbook suggests batching those lookups to cut the cost.

Plain hashing picks a machine from the remainder of the hash divided by the machine count. Add a machine and most hosts move. Consistent hashing places machines on a ring, so only a share of hosts moves.

We measured it on 100,000 made-up hostnames, growing from 3 to 4 machines. Plain hashing moved 75% of hosts. Consistent hashing, with 100 ring points per machine, moved 29%, close to the ideal 25%.

In the diagram, notice that every link, from any machine, passes through the splitter. That one rule keeps each host on a single machine.

A crawler handles traps with limits, failures with checkpoints, and recrawling with change history. None of these is clever, but leaving any one out breaks a long crawl.

Crawler traps

A crawler trap is a site structure that produces endless URLs, like the calendar running to the year 2400. Some traps are accidents, and some are built on purpose.

• Cap URL length and path depth, because trap URLs tend to keep growing.
• Cap pages per host per crawl, so one site cannot eat the whole budget.
• Watch for paths that repeat the same segment again and again.
• Slow down on hosts whose pages keep matching existing fingerprints, a sign of empty generated pages.

Failures

Fetchers will crash, and a crawl runs for weeks. The team keeps the frontier in a durable store and checkpoints its in-memory parts every few minutes. A restarted machine reloads its hosts and carries on.

Failed fetches get retries with growing waits. After three failures, a URL is parked until the next crawl, and its host's gap grows.

Recrawling

Recrawling means fetching known pages again to catch changes. Pages that changed on each of the last few visits move to the hot queue, and pages that never change cool down.

Conditional requests save more. The crawler sends the date of its last copy in an If-Modified-Since header. A server that supports it replies 304 Not Modified, which must not carry a body.

Lead with requirements and scale, draw the basic loop, then spend most of your time on the frontier, politeness and de-duplication. Those three are where follow-up questions usually go.

1. 1

   Ask the scope questions

   Ask how many pages, how fresh, which content types, and whether JavaScript must run. Agree that indexing and ranking are out of scope.

2. 2

   Estimate out loud

   Work out pages per second, storage per crawl and seen-URL memory. Then show the politeness sum for one big site.

3. 3

   Draw the basic loop

   Seeds, frontier, fetchers, page store, parser, duplicate checks, and back to the frontier. Keep it to one row of boxes.

4. 4

   Open up the frontier

   Show front queues for priority, back queues holding one host each, and a heap of next allowed times.

5. 5

   Add politeness rules

   Check robots.txt just before fetching, reuse it for at most 24 hours, keep a gap per host, and back off on 429 or 503 replies.

6. 6

   Explain both duplicate checks

   Normalise URLs and test a seen set or Bloom filter before fetching. Fingerprint page bodies after fetching.

7. 7

   Scale out and fail safely

   Hash hosts to machines, split fingerprints by value, checkpoint the frontier, and guard against traps.