Design a Price Tracker: Polling, Alerts and Caching

A simple price alert app breaks because it checks every user's request separately, on one fixed timer, with no limit per store. Growth multiplies the requests until stores block it and alerts arrive late.

Picture a common setup. A small team runs a price-drop alert app for electronics. You paste a product link, set a target price, and the app tells you when the price falls to it. The setup is made up, but the problems are real.

The first version was a cron job, a timer that runs a script on a schedule. It woke every hour, looped over every watch, fetched the product page and compared the price.

A watch here means one user's request to track one product below one target price. In the database it is just one row: a user, a product and a number.

That worked for 2,000 users. A year later the app had 2.4 lakh (240,000) users and 6 lakh watches across five online stores. Then the Diwali sale week arrived, and four things went wrong.

• The hourly job took almost three hours to finish, so a sale price could end before anyone heard about it.
• One store started replying with 429, the code that means too many requests, and then blocked the app's servers for a day.
• A pricing glitch showed a Rs 64,999 laptop at Rs 649 for six minutes. About 40,000 users got an alert for a deal that was gone.
• Some users got the same alert four times, because the job crashed and restarted halfway.

People forgive a late alert once, but a fake one twice makes them delete the app. The rest of this guide rebuilds the app around smart polling, caching and careful alerts.

A price tracker system design needs to fetch prices on a schedule, notice changes, keep a price history, and notify users whose target price is reached. It must do this without overloading the stores it reads from.

What the app must do

• Let a user add a product link and a target price, and remove it later.
• Check the current price of every watched product on some schedule.
• Show a price history chart for each product.
• Send an alert by push notification or email when the price reaches the target.

How well it must do it

• Freshness: popular products are checked every 15 minutes, and quiet ones at least once a day.
• Alert speed: an alert reaches the user within a few minutes of the change being seen.
• Correctness: hold back glitch prices, and never send the same alert twice.
• Politeness: no store gets more requests than its budget allows.

Now do quick sums with the app's numbers. The 6 lakh watches point at only 80,000 unique products, because many people want the same phones and earbuds. That one fact changes the whole design.

Checking every watch hourly means 6 lakh fetches an hour, or 1.44 crore (14.4 million) a day. Checking every unique product hourly means 80,000 an hour, or 19.2 lakh a day. Deduplication, removing repeats, cuts the work 7.5 times.

Here is the shape of the new design. Follow the arrows from the app to the alert at the end. Notice that users never trigger a fetch directly: the scheduler owns all traffic to stores.

A price tracker should check each product about as often as its price tends to change, not on one fixed timer. Busy products get short gaps between checks, and quiet products get long ones.

Polling means asking the source again and again whether something changed. It is simple and works with any store. Its cost is that most checks find nothing new, so every wasted check should be cheap or skipped.

Think of an aunty in Ahmedabad with a stack of raddi (old newspapers) on the loft. She asks the kabadiwala, the scrap dealer, his rate per kilo every week. When the rate starts moving, she asks every few days. When it sits still for months, she asks less.

She also has one number in mind. She sells only when the rate crosses it, not every time it twitches. In the tracker, her asking is a fetch, the time between asks is the gap, and her number is the target price.

Where the picture breaks: she watches one rate from one dealer. The tracker watches 80,000 prices, and each store also sets how often it may be asked at all.

Adaptive polling in plain words

Adaptive polling means each product carries its own gap, the time until its next check. In this design, a change halves the gap, down to a floor of 15 minutes. No change stretches it by half, up to a ceiling of one day.

Starting from one hour, eight quiet checks in a row take a product to the one-day ceiling. Two changes in a row take it to the 15-minute floor. The floor and ceiling stop runaway values.

A few rules can also move a product into a faster lane. A product with many watchers matters to more people. A price close to many targets deserves a closer look. For sizing, the team groups products into three tiers.

The tiers make 12.8 lakh fetches a day, about 11 times fewer than version one. Hot products are still checked four times more often than before.

Spread over a day, that is about 15 fetches per second, or roughly 3 per second per store. Sale peaks run higher, so the design must cap each store separately.

You keep one sorted list of products ordered by their next check time, and workers keep taking the ones that are due. Nothing loops over the whole catalogue, so the work stays smooth all day.

Version one had a burst problem. The cron job fired on the hour, sent requests as fast as it could, and then sat idle. A sorted schedule spreads checks across the clock, because each product's next time is different.

Redis, a fast data store that keeps its data in memory, has a type made for this called a sorted set. It stores unique members, each with a number called a score, and keeps them ordered by score.

Here the member is a product id and the score is its next check time, as a Unix timestamp (seconds since 1 January 1970). Adding a member costs O(log N) time, which grows very slowly as the set grows.

You can see the idea in two commands with redis-cli, the Redis command-line tool. Pretend the time now is 1000: p1 and p2 are due, and p3 is far in the future.

redis-cli ZADD next_check 100 p1 200 p2 9999999999 p3
redis-cli ZRANGE next_check -inf 1000 BYSCORE LIMIT 0 500

The first command reports 3, the number of members added. The second lists p1 and p2, the due products, oldest first. LIMIT 0 500 caps one batch at 500 products, so a backlog is worked through in slices.

After each check, the worker writes the product's new next check time with ZADD, which moves it to its new place in line. The engineer section shows the full code, and the trap in letting several workers read one list.

You poll stores politely: use an official feed if there is one, follow robots.txt, cap requests per store, slow down when a store says so, and use conditional requests. A blocked fetcher checks nothing at all.

Start with the cleanest source. Ask whether a store offers a product feed or an API, a way for programs to request data, through a partner or affiliate programme. A feed is a file or endpoint that lists products and prices in a fixed format, so nothing breaks when the page design changes.

Robots.txt rules and what the standard says

When you do read public pages, check robots.txt first. It is a plain text file at the top of a site, at the address ending in /robots.txt, listing which paths bots may visit. RFC 9309, published in September 2022, is the standard that defines it.

• Freshness: crawlers should not use a cached robots.txt for more than 24 hours, unless the file is unreachable. So the tracker fetches it again at least daily.
• Server errors: if robots.txt returns a 5xx code, a server error, the crawler must assume it may crawl nothing. After a long outage, such as 30 days, it may treat the file as missing or keep a cached copy.
• Missing file: if it returns a 4xx code, such as 404 Not Found, the crawler may access anything.
• Not a lock: the standard says these rules are not a form of access authorization. Following them is good manners, so also read each store's terms of use.

A request budget for each store

Each store gets its own budget, such as 5 requests per second, enforced by a rate limiter in front of the fetchers. A rate limiter is a gate that lets through only a set number of requests per second.

When a store replies 429 Too Many Requests, the fetcher slows down for that store only. RFC 6585 defines 429 and says the reply may include a Retry-After header saying how long to wait.

Retry-After comes in two forms: a number of seconds, such as 120, or a date, such as Wed, 21 Oct 2015 07:28:00 GMT. Handle both. If there is none, wait your own default before trying again.

Google's crawler reads these signals the same way. Google's guide for site owners says many 500, 503 or 429 replies make Googlebot, its crawler, slow down for the whole hostname, the site address such as shop.example.com. So a store sending 429 is asking you to back off.

Conditional requests make unchanged pages cheap

A conditional request asks the server to send the page only if it changed since your copy. If nothing changed, the server replies 304 Not Modified with no body, so almost nothing travels.

It works through an ETag, a short fingerprint the server sends with each page. The tracker keeps the ETag in its cache. Next time it sends it back in an If-None-Match header. A server that supports this returns 304 when the page has not changed.

You can watch this happen with curl, a command-line tool for web requests. The script below uses a public package registry page because it sends an ETag; swap in any URL that does. You do not need every symbol: it saves the ETag, then asks again with it.

URL=https://registry.npmjs.org/left-pad
ETAG=$(curl -sI "$URL" | tr -d '\r' | awk -F': ' 'tolower($1)=="etag"{print $2}')
echo "ETag: $ETAG"
curl -s -o /dev/null -w "first: %{http_code}, %{size_download} bytes\n" "$URL"
curl -s -o /dev/null -w "again: %{http_code}, %{size_download} bytes\n" -H "If-None-Match: $ETAG" "$URL"

When we ran it on 19 October 2026, the first request returned 200 with 22,573 bytes and the second returned 304 with 0 bytes. Your ETag and byte count may differ; the 304 and the 0 bytes are what to look for.

Not every store sends ETags, and some pages change on every load because of ads. For those, the fetcher hashes only the price block it extracts. If the hash matches the last one, nothing downstream runs.

The diagram shows one check from start to finish. Look at the two question boxes: the first protects the store, and the second decides how much of your own system wakes up.

Caching cuts cost by answering repeated questions from fast memory instead of the store or the main database. A price tracker caches in three places: the latest price, the page fingerprint, and the app's chart reads.

A cache is a small, fast copy of data you read often. The trade-off is staleness: a cached value can be out of date. The trick is to decide, for each cached thing, how stale is acceptable.

• Latest price cache. One small record per product holds the last price, when it was seen, the ETag and the current gap. The app reads from here, so opening a product never triggers a store fetch.
• Fingerprint cache. The ETag or price hash lets a fetcher stop early when nothing changed.
• Chart cache. Price charts are read far more than they change. Each product's 90-day chart is cached and rebuilt only when a new price arrives.

The biggest saving is the rule that users never fetch. Say 3,000 people open the same phone page during a sale. They all read one cached price, and the store sees only the fetch the scheduler planned.

Storing price history without wasting space

Store a history row only when the price changes, not on every check. Most checks find the same price, so this keeps the history table small and charts quick to draw.

Each row holds a product id, a time and a price in paise, the hundredth part of a rupee. Paise are whole numbers, so you avoid rounding bugs from decimals.

For long charts, keep a daily summary per product: lowest, highest and closing price. A two-year chart then reads 730 rows, not every change. If the table grows large, partitioning it, which means splitting it into parts, keeps queries fast.

Price drop alerts reach users quickly when every real price change becomes an event, and an alert worker matches that event against the product's target prices. Only users whose target is crossed are notified.

An event here is a small message saying, for example, product 4471 moved from Rs 21,999 to Rs 18,499 at 9:42 p.m. The fetcher puts it on a message queue, a buffer that holds messages until a worker reads them.

The queue matters during sales. If a thousand prices change in one minute, the queue absorbs the burst. Alert workers read at their own pace, and the fetchers never wait for notifications to send.

To find the right users fast, keep each product's watches sorted by target price. A new price then needs one binary search, a way to find a spot in a sorted list by halving it repeatedly. Everyone from that spot upward has a target at or above the new price.

Read the diagram top to bottom. Each box is one step, and its detail line says which part talks to which.

Sending is its own job. A notification service handles push messages, email retries and each user's daily cap. Here the cap is three alerts per user per day, grouped if needed, so a sale does not flood anyone's phone.

You stop false alerts by confirming large or strange drops before sending them. Fetch the price again after a short delay, check stock and seller, and hold back anything that looks like a glitch.

The Rs 649 laptop, a 99 percent drop, taught the team that a correct reading can still be a bad alert. The page really showed that price. It just did not last, and nobody could really buy it.

• Confirm big drops. Any drop over 40 percent waits about ten minutes, longer than the Rs 649 glitch lasted, and is fetched again. A longer glitch can still pass, so the checks below matter too.
• Check stock. A low price on an out-of-stock item is not a deal, so the fetcher also reads availability.
• Check the seller. Some stores list many sellers on one product page, so compare prices from the same seller only.
• Separate list price from coupon price. A price that needs a bank card offer is shown as such, not as the main price.
• Watch your own parser, the code that reads the price off the page. When a store changes its layout, the parser may read the wrong number.

That last one is easy to miss. If 2,000 products from one store all drop by 90 percent in an hour, the store did not run a magic sale. Your parser broke, and alerts for that store should pause automatically.

During a big sale, prices change far more often and users open the app far more, so polling and alerts spike together. A good design absorbs this with queues, per-store limits and caches, not more raw fetching.

• Fetchers can scale out, but each store's budget stays the same. Extra workers just wait their turn inside that budget.
• The price change queue grows during bursts and drains after. Alert delay rises from seconds to a few minutes, which this design accepts.
• The app reads only cached prices, so a traffic spike in the app never reaches any store.

At the next big sale, alerts arrived in about four minutes on average and no store blocked the app. The confirmation step held back three glitch prices (all made up).

You make reading the due list and claiming it one atomic step, so no other worker can see those products in between. In Redis, a short script in Lua, a small language Redis can run, does this. Redis runs each script atomically: nothing else runs on the server until it finishes.

The trap is easy to write. Each worker reads the due products, fetches them, and only then writes new times. While it fetches, other workers read the same list and fetch the same products.

The script below finds up to 500 due products and pushes each one's score five minutes ahead, which is the lease. If the worker dies mid-fetch, the lease runs out and the product becomes due again.

Pick a lease longer than your slowest fetch, including any wait for the store budget. Otherwise a second worker claims the product while the first is still busy with it.

import time
import redis

r = redis.Redis(decode_responses=True)

CLAIM = r.register_script("""
local due = redis.call('ZRANGE', KEYS[1], '-inf', ARGV[1],
                       'BYSCORE', 'LIMIT', 0, ARGV[2])
for _, id in ipairs(due) do
  redis.call('ZADD', KEYS[1], ARGV[3], id)
end
return due
""")

def claim_due(limit=500, lease=300):
    now = time.time()
    return CLAIM(keys=["next_check"], args=[now, limit, now + lease])

def reschedule(product_id, changed):
    gap = float(r.hget("gap", product_id) or 3600)
    if changed:
        gap = max(gap / 2, 900)      # floor: 15 minutes
    else:
        gap = min(gap * 1.5, 86400)  # ceiling: one day
    r.hset("gap", product_id, gap)
    r.zadd("next_check", {product_id: time.time() + gap})

We ran it with redis-py 8.1.0 against Redis server 7.0.15. Two claims in a row returned the due products once, then an empty list. Each claimed product's score moved 300 seconds ahead.

Then eight worker processes shared 10,000 due products, with a 10-millisecond pretend fetch per batch of 500. The plain read-then-write version fetched them between 59,000 and 68,000 times over four runs. The claim script fetched each exactly once.

Two version notes, as of 19 October 2026. ZRANGE with BYSCORE needs Redis 6.2 or later. The older ZRANGEBYSCORE command still works, but Redis has marked it deprecated since 6.2.0.

It depends on your Python version, as of 19 October 2026. In our test cases, Python 3.13.16 followed the standard's matching rules, while 3.11.17 and 3.12.3 did not.

RFC 9309 says the most specific rule wins, meaning the one that matches the most characters. It also says crawlers must understand two wildcards: * for any characters, and $ for the end of the path.

Here is the test. One rule blocks /shop/, a longer rule allows /shop/deals/, and a third blocks any path with ?sort= in it.

from urllib.robotparser import RobotFileParser

rules = """
User-agent: *
Disallow: /shop/
Allow: /shop/deals/
Disallow: /*?sort=
"""
rp = RobotFileParser()
rp.parse(rules.splitlines())
for path in ["/shop/deals/phone-123", "/shop/cart", "/phones?sort=price"]:
    print(path, rp.can_fetch("PriceBot", path))

On 3.11.17 and 3.12.3, the first matching rule wins, and a * inside a path is not treated as a wildcard. That blocks a deals page the store wanted crawled, and crawls sorted pages it asked you to skip.

Protego, a robots.txt parser on PyPI (Python's package index) that says it implements RFC 9309, gave the standard's answers. Its can_fetch takes the URL first and the bot name second, the reverse of Python's.

Two status cases need your own rule. A 401 or 403 makes Python block everything, which is stricter than the standard requires. A 429 makes it allow everything, though 429 means slow down; Google's crawler documents 429 as an exception to its 4xx rule.

The change shipped in Python 3.13.14 and 3.14.5, listed as gh-138907 in the changelogs. As of 19 October 2026, earlier 3.13 and 3.14 releases and every 3.11 and 3.12 release use first match. Run the snippet on the exact Python you deploy.

You record each user's last alert price in a table with a unique key, and claim the alert in one statement before sending it. A replayed event then finds the claim already made and sends nothing.

This matters because a crash can replay an event, as the four-times alert showed. The rule here: alert a user again only when the price goes below the last price they were told.

import sqlite3
from bisect import bisect_left

db = sqlite3.connect("alerts.db", timeout=30)  # wait up to 30 s for the write lock
db.execute("""CREATE TABLE IF NOT EXISTS last_alert (
    user_id INTEGER, product_id INTEGER, price_paise INTEGER,
    PRIMARY KEY (user_id, product_id))""")

def users_to_alert(targets, new_price):
    # targets: sorted list of (target_paise, user_id) for one product
    start = bisect_left(targets, (new_price,))
    return [user_id for _, user_id in targets[start:]]

def claim_alert(user_id, product_id, price):
    # True only if this user has not been told about this price or lower
    cur = db.execute("""
        INSERT INTO last_alert VALUES (?, ?, ?)
        ON CONFLICT (user_id, product_id) DO UPDATE
        SET price_paise = excluded.price_paise
        WHERE excluded.price_paise < last_alert.price_paise""",
        (user_id, product_id, price))
    db.commit()
    return cur.rowcount == 1

The ON CONFLICT clause turns the insert into an update when the user and product already have a row. The update happens only for a lower price, and rowcount says whether this call won.

We ran it with Python 3.13.16 and SQLite 3.45.1, a small file-based database. The ON CONFLICT form needs SQLite 3.24.0 or later. With targets of Rs 18,000, 18,499, 19,000 and 20,000, an event at Rs 18,499 alerted the last three users. Replaying it alerted nobody.

A rise to Rs 19,499 alerted nobody, and a new low of Rs 17,999 alerted all four. SQLite lets only one process write at a time, so the 30-second timeout makes other processes wait their turn.

With eight processes replaying one event for 1,000 users, exactly 1,000 claims won in all 20 runs. With Python's default wait, most processes crashed with a database is locked error. In one run on a busy machine, seven processes still gave up after 30 seconds, so retry when SQLite reports the database is locked.

Two edges follow from the re-alert rule, and we tested both. After a rise to Rs 21,999 and a later sale back at Rs 18,499, nobody was alerted again. A one-rupee drop to Rs 18,498 alerted all three again. So delete a user's row when the price rises above their target, and consider asking for a minimum further drop, such as 2 percent.

One choice remains: claim before sending, or after. Claim first, as here, and a crash between the two loses that one alert. Send first, and a crash can repeat it. When the claim and the send live in different systems, no ordering gives exactly once across a crash.

Explain it as three loops: a polling loop that fetches prices, a storage loop that records changes, and an alert loop that notifies users. Then show how each loop stays cheap and safe.

1. 1

   Clarify the scope

   Ask how many users, watches and stores, how fresh prices must be, and whether stores offer feeds. Confirm alerts go by push and email.

2. 2

   Estimate the work

   Show that watches far outnumber unique products. Work out fetches per second per store with a simple tiered schedule.

3. 3

   Design the scheduler

   Use a sorted set keyed by next check time and adaptive gaps. Explain how workers claim products atomically, with a lease for crashed workers.

4. 4

   Make fetching polite

   Mention feeds first, robots.txt, per-store rate limits, backing off on 429 with Retry-After, and conditional requests with ETags.

5. 5

   Store only changes

   Keep the latest price in a cache and write history rows only on change, with daily summaries for long charts.

6. 6

   Design the alert path

   Publish change events to a queue, match targets with a sorted list, claim each alert once, and cap alerts per user per day.

7. 7

   Cover failure cases

   Confirm big drops, check stock and seller, and pause a store's alerts when its parser breaks.