Design an Ad Click Prediction System: CTR, Calibration, Auctions

Ranking by bid alone can fill slots with costly ads that few readers click, and ranking by an ad's average past click rate hides who clicks it and where. Both rules lose money for the network and the advertisers.

Picture a common setup. An ad network places ads inside regional-language news apps in Hindi, Marathi, Tamil and Bengali. Between paragraphs of a story sits an ad slot. The setup is made up, but the problem is real.

The apps send about 4 crore (40 million) slot requests a day. For each one there are around 300 eligible ads, from gold loan lenders to tractor dealers. Each advertiser sets a bid: the most it will pay for one click.

A naive starting rule takes each ad's past click-through rate, or CTR, and multiplies it by the bid. Brand-new ads have no past, so they get a default CTR. This rule fails in three ways.

• A tractor dealer's ad does well in Marathi farming stories but badly in Bengali cricket stories. One average CTR hides that difference.
• A new jewellery advertiser gets almost no views in its first week, because the default CTR is low. It never gets the chance to prove itself.
• A reader sees the same loan ad five times in one session. Nothing in the rule knows they already ignored it four times.

Each failure has the same cause. The rule asks how good an ad is on average, when the real question is about this reader, this story and this slot. Ad click prediction replaces the rule with a model that answers that question.

Ad click prediction is a machine learning task that outputs the probability that a given reader clicks a given ad in a given context. That probability is called the predicted CTR, or pCTR.

A model here is a function learned from past data. You feed it facts about one view, and it returns a number between 0 and 1. A pCTR of 0.004 means the model expects about 4 clicks per 1,000 views like this one.

Clicks are rare. In the running example about 4 views in 1,000 end in a click (made up). With so few clicks, small errors in the model become large errors in money, as the next two sections show.

Here is one picture to hold on to. A trader at the Pushkar camel fair is deciding what a camel is worth to him. He estimates the chance he can resell it and the price he would get.

Where the picture breaks: the trader decides once, with his own money. An ad network runs this sum crores of times a day, and the price the winner pays depends on the runner-up, not just on its own number.

The auction multiplies each ad's bid by its pCTR to get the expected money per view, then shows the ad with the highest value. Quoted per 1,000 views, this is the eCPM (effective cost per mille, where mille means thousand).

Here is one slot in a Marathi news app, inside a story about crop prices. Three ads are eligible. Look at the tractor dealer: it has the highest bid but does not win.

The gold loan ad wins because readers of crop-price stories click it often. Ranking by bid alone would show the tractor ad, and the expected value of this view would drop from Rs 0.040 to Rs 0.030.

How much does the winner pay?

In a second-price style auction, the winner pays just enough per click to keep its rank, not its full bid. That price is the runner-up's value per view divided by the winner's own pCTR.

To beat the tractor ad's Rs 0.030 per view, the gold loan ad needs 0.030 divided by 0.005, which is Rs 6 per click, not Rs 8.

Search ads have long used a version of this called the generalized second-price auction, or GSP. An economics paper by Edelman, Ostrovsky and Schwarz showed that in GSP, bidding your true value is not an equilibrium, a set of bids nobody wants to change.

That sets GSP apart from VCG (Vickrey-Clarke-Groves), the textbook auction it looks like, where bidding your true value is always a safe strategy. The two look alike on paper but reward bidders differently.

Some ad exchanges, the marketplaces where many buyers bid for the same slot, run first-price auctions instead, where the winner pays its own bid. OpenRTB 2.6, the industry standard for these bid messages, has a field named at for this. Value 1 means first price, and 2 means second price plus.

Now notice where pCTR sits in the price formula. If the model wrongly says 1% instead of 0.5% for the gold loan ad, its price per click halves to Rs 3. Readers still click 0.5% of the time, so the network earns Rs 0.015 per view instead of Rs 0.030.

The error also runs the other way. An over-predicted runner-up raises the price the winner pays. A wrong pCTR does not just reorder ads; it changes what advertisers pay.

Calibration means predicted probabilities match what really happens: if the model says 1% for a group of views, about 1% of them get clicked. A model can rank ads well and still be badly calibrated.

A common way to measure this is the calibration ratio: total predicted clicks divided by total real clicks. A 2014 industry paper on ad click prediction (He and colleagues, ADKDD 2014) defines it this way: the closer to 1, the better.

A ratio of 1.6 means the model expects 60% more clicks than really happen. A ratio of 0.8 means it expects 20% too few.

Ranking quality is a separate thing. AUC is a ranking score: the chance that a randomly picked clicked view gets a higher score than a randomly picked unclicked one. It only looks at order, never at the numbers themselves.

Pause and predict: if you double every prediction, what happens to AUC and to the calibration ratio? Run the check below to see. It needs only Python 3, no libraries.

def calibration_ratio(predicted, clicked):
    return sum(predicted) / sum(clicked)    # predicted clicks / real clicks

def auc(scores, clicked):
    pos = [s for s, c in zip(scores, clicked) if c]
    neg = [s for s, c in zip(scores, clicked) if not c]
    wins = sum((p > n) + 0.5 * (p == n) for p in pos for n in neg)
    return wins / (len(pos) * len(neg))

scores = [0.1, 0.2, 0.3, 0.4]    # toy pCTRs for four views
clicked = [0, 0, 1, 0]           # only the third view was clicked
doubled = [2 * s for s in scores]
print(round(auc(scores, clicked), 3), round(auc(doubled, clicked), 3))    # 0.667 0.667
print(round(calibration_ratio(scores, clicked), 2),
      round(calibration_ratio(doubled, clicked), 2))                       # 1.0 2.0

What just happened? AUC stays at 0.667 for both lists, while the calibration ratio jumps from 1.0 to 2.0. The order of the views is the same; the numbers are not.

The same thing held on a larger test. On 2 million simulated views with scikit-learn 1.7.2, doubling every prediction left AUC at 0.7098 and moved the calibration ratio from 1.00 to 2.00.

Why one overall ratio can hide a bad group

Suppose the network launches a new model that has the best AUC so far. For ads less than a week old, its calibration ratio is 1.6, because new ads have few history features and the model fills the gap with optimism (made up).

The overall ratio reads 1.02, which looks perfect, because new ads are a small share of views and older ads sit at 0.97. Meanwhile new ads win slots they do not deserve, at prices per click that are too low.

Beginners can stop here. You now know what pCTR is, how the auction uses it, and why its numbers must be honest. The rest of the guide is for engineers who build and run such a system.

An ad click prediction system has two halves: an offline part that learns from past views and clicks, and an online part that scores ads for each live request. The trained model file is the bridge between them.

This diagram shows the online half, the path one slot request takes. Follow the arrows from the news app to the chosen ad. Notice that the calibrator sits between the model and the auction.

The candidate filter is plain code, not a model. It drops ads whose budget is spent, whose language does not match, or whose targeting excludes this reader. Scoring 60 ads is far cheaper than scoring 300.

The impression log matters more than it looks. An impression is one ad shown once. Every logged impression becomes a training example later, once you know whether it was clicked.

A CTR model uses facts about the reader, the ad, the context and their past interactions. A feature is one such input fact. In the 2014 paper, the ten most important features were all historical ones, such as past clicks by the user or on the ad.

• Reader: app language, district, phone type, and topics the reader opened this week.
• Ad: advertiser, ad category such as gold loans, the image style, and the ad's recent CTR.
• Context: story topic, slot position in the story, hour of day, and whether it is a festival week.
• History: how many times this reader saw this ad today, and how often they clicked this category before.

History features fix the third failure of the naive rule. Once the model knows a reader saw the loan ad four times without a click, it can predict a much lower chance for view five.

The same paper notes that contextual features still matter for cold start: new users and new ads have no history yet, so context is all the model has.

Cross features and the hashing trick

A cross feature joins two facts into one, such as language plus ad category. Marathi plus tractors behaves very differently from Bengali plus tractors. That cross fixes the first failure of the naive rule.

Crosses create millions of possible values, too many to list in advance. A common answer is the hashing trick: a hash function turns any text into a number, and that number picks a slot in a fixed-size table of weights.

Here is a small scorer for logistic regression, the classic CTR baseline. It adds one learned weight per feature, then squeezes the total into a probability with the logistic (sigmoid) function. The toy weights let you run it.

import math
import zlib

NUM_BUCKETS = 2 ** 22    # toy size; size a real table by measuring collision loss

def bucket(name, value):
    key = f"{name}={value}".encode()
    return zlib.crc32(key) % NUM_BUCKETS    # same slot in every process

def predict_raw(weights, bias, features):
    z = bias
    for name, value in features.items():
        z += weights.get(bucket(name, value), 0.0)    # unseen feature adds 0.0
    return 1 / (1 + math.exp(-z))    # squeeze into 0 to 1

# Toy weights for the check; a real trainer learns millions of them.
weights = {bucket("lang", "marathi"): 0.3,
           bucket("lang_x_category", "marathi|gold_loan"): 0.8}
bias = -6.4
view = {"lang": "marathi", "ad": "A1043", "slot": "para_3",
        "lang_x_category": "marathi|gold_loan"}
print(round(predict_raw(weights, bias, view), 4))    # 0.005
print(bucket("lang", "marathi"))                     # 6864

Two lines matter most. The bucket function uses crc32, not Python's built-in hash. Python randomises hash() of str and bytes per process by default, unless PYTHONHASHSEED is set to a fixed integer. A trainer and a server would then disagree on slot numbers.

You can see this for yourself in a terminal. The first loop prints two different numbers, and the second prints 6864 twice (checked on Python 3.13).

for i in 1 2; do python3 -c "print(hash('lang=marathi'))"; done
# two different numbers: str hashes are randomised per process
for i in 1 2; do python3 -c "import zlib; print(zlib.crc32(b'lang=marathi') % 2**22)"; done
# 6864 twice

Unseen features fall back to a weight of 0.0, so a brand-new ad id adds nothing instead of crashing. The ad and slot features above have no toy weight, which is why the output is 0.005.

Hashing has a cost. Two different features can land in the same slot, a collision, and then share one weight. A 2013 industry paper, Ad Click Prediction: a View from the Trenches, reports that aggressive hashing gave them no useful memory savings.

They could not shrink below several billion features without a visible loss in accuracy. Treat hashing as a convenience for unbounded crosses, and measure the table size you need instead of assuming small is free.

Start with logistic regression on hashed features, because it is fast, cheap and easy to debug. Move to tree-based or deep models only when offline tests and a live experiment both show a real gain.

The 2014 paper combined boosted decision trees with logistic regression. Boosted trees are many small yes-or-no rule trees, each fixing the last one's mistakes. Each tree sorts a view into one leaf, and that leaf becomes a yes-or-no feature for the logistic regression on top.

The authors report this beat either method alone by over 3% on their main metric, normalized entropy, explained in the measuring section. That is one dataset and one metric, not a universal law.

Both classic papers report that per-coordinate learning rates help online logistic regression. Each feature gets its own learning rate, which shrinks as that feature builds up training updates.

The 2014 paper found this the most accurate of the learning-rate schemes it tested. The 2013 paper reports it cut AucLoss, which is 1 minus AUC, by 11.2% against one global rate.

Deep models learn crosses by themselves instead of needing them written by hand. They often score better but cost more to run and are harder to debug. A safe plan has three stages.

1. 1

   Ship a logistic regression baseline

   Hashed features plus a few dozen hand-written crosses. Measure its scoring time for all candidates on one CPU core before you add anything heavier.

2. 2

   Prove the pipeline first

   Logging, label joining, training, calibration and monitoring must work end to end. A fancy model on a broken pipeline just fails faster.

3. 3

   Try a deep model behind an experiment

   Send a small share of traffic to it. Keep it only if revenue per thousand views rises without hurting calibration or speed.

You build training data by joining the impression log with the click log on a shared impression id. An impression with a matching click is a positive example, and one without is a negative.

The hard part is time. A click can arrive seconds or minutes after the view. The running example waits 30 minutes before calling an impression unclicked (made up).

The 2014 paper describes this trade-off for its online joiner, the service that pairs views with clicks as they stream in. Too short a waiting window loses clicks that arrive late, and too long a window delays fresh training data. You must balance recency against click coverage.

Freshness matters. In the 2014 paper, training daily instead of weekly cut normalized entropy by about 1% for both models they tested. The 2013 paper goes further and trains continuously with online learning, updating the model as each labelled view arrives.

Negative downsampling and how to undo it

Since clicks are rare, teams often keep every click but only a small share of non-clicks for training. This is negative downsampling, and it makes training much cheaper.

The catch is that the model learns from a world where clicks look common. Keep 5% of non-clicks, and a true CTR of 0.4% looks like about 7.4% in the training data. The model's raw outputs are too high by the same amount.

The 2014 paper gives the correction: q equals p divided by (p plus (1 minus p) divided by w). Here p is the raw output and w is the share of non-clicks kept. The check below runs it both ways.

def undo_downsampling(p, keep_rate):
    # p: raw model output; keep_rate: share of non-clicks kept, e.g. 0.05
    return p / (p + (1 - p) / keep_rate)

true_ctr, keep_rate = 0.004, 0.05
seen = true_ctr / (true_ctr + (1 - true_ctr) * keep_rate)    # CTR in the sample
print(round(seen, 4))                                  # 0.0743
print(round(undo_downsampling(seen, keep_rate), 4))    # 0.004

With w of 0.05, a raw 7.4% becomes about 0.4%. Forgetting this one line would make every pCTR about 18.6 times too high at this click rate.

The 2013 paper used a different fix. It kept only a share r of search queries with no clicks, and gave each kept one an importance weight of 1 divided by r. Each kept query then counts as 1/r queries in training, so the model learns the true rate directly.

We ran both fixes on 2 million simulated views with scikit-learn 1.7.2, keeping 5% of non-clicks. The raw model's calibration ratio was 17.69. The formula brought it to 0.999, and importance weights to 1.023. These numbers score the same views the sample came from; on 2 million fresh simulated views, the formula's ratio was 0.974.

How much to drop is a tuning choice. The 2014 paper found the rate clearly changes accuracy, and reports its best result at 0.025, a value missing from the list of rates it says it tested. Test a few rates on your own data rather than copying any one number.

Fitting a calibrator after training

You can also fit a small calibrator on fresh data after training. Platt scaling fits a logistic curve to the raw scores. Isotonic regression fits a staircase that never goes down, so it fixes the numbers without reversing the order, though it can merge close scores into ties.

The 2013 paper describes such a calibration layer and requires the mapping to be isotonic, meaning it only goes up. A 2017 study of neural network classifiers found temperature scaling, a one-number version of Platt scaling, surprisingly effective on most of their datasets.

This diagram shows the offline half. Start at the two logs on the left and follow them to the model registry, the store that keeps every approved model version.

The feedback loop that hides mistakes

One more trap hides in this data. The model only learns from ads the auction chose to show. If it wrongly gives a new ad a low pCTR, the ad never shows, and the model never learns it was wrong.

The usual fix is exploration: a small share of slots shows an eligible ad that did not win. The running example uses 2% (made up). That answers the second failure of the naive rule, because new ads get a fair first look.

Exploration has a price, since those slots earn less than the winner would. Keep the share small, log which impressions were exploration, and use them to check calibration on ads the model rarely shows.

You serve CTR predictions fast by filtering candidates cheaply, keeping features in memory, and scoring all candidates in one batch. Every step gets a fixed slice of the time budget.

The time limit is real. OpenRTB 2.6 is still the current 2.x spec as of 19 October 2026. Its bid request carries tmax: the most milliseconds the exchange allows for bids to arrive, including network time. A late answer simply misses the auction.

In the running example, the news apps wait 80 ms for an ad before showing the story without one (made up). The budget splits across the steps like this.

• Network to and from the app: about 40 ms, which the ad team cannot change.
• Candidate filter: 5 ms, using an index of ads by language and topic.
• Feature lookup: 10 ms, from an in-memory key-value store holding reader and ad counts.
• Model scoring: 5 ms for all 60 candidates together, not one by one.
• Calibration and auction: 2 ms, and whatever is left is safety margin.

A feature store, a system that serves the same feature values to training and serving, keeps both sides consistent. If training counted views per day but serving counts views per hour, the model reads numbers it never learned from.

You measure it offline with log loss, AUC and calibration on held-out data, then online with a live experiment on revenue, clicks and advertiser outcomes. Offline wins do not always survive real traffic.

Log loss punishes confident wrong guesses heavily, so it rewards both good ranking and honest numbers. In our simulated test, doubling every prediction raised log loss from 0.02516 to 0.02641 while AUC did not move.

Normalized entropy, the 2014 paper's main metric, is the model's average log loss divided by the log loss of always predicting the average CTR. Below 1 means the model beats that guess; our simulated model scored 0.9534.

The 2013 paper adds a practical habit: compare relative changes between models on the same data, not absolute values. Absolute numbers shift with traffic mix, while relative changes stay much more stable over time.

Online, run an A/B test: part of the traffic sees the old model and part sees the new one. Watch revenue per thousand views, CTR, calibration per group, and how many advertisers spend their full budget.

Offline checks run before launch, so they cannot catch drift in live traffic. An hourly calibration guard closes that gap. Look at the question box in the middle of this diagram.

The 0.9 to 1.1 band and the 3-hour fallback are made-up starting points. Clicks per hour in a small group can be tiny, so set a minimum, such as 200 clicks in the window, before the ratio counts.

Explain it as a money problem first and a model problem second. Show how pCTR flows into ranking and pricing, then design training and serving around that.

1. 1

   Clarify the business

   Ask how advertisers pay: per click, per view or per action. Ask the request volume and the time limit per request.

2. 2

   Write the ranking formula

   Rank by bid times pCTR. Explain that pCTR also sets the price, so it must be a calibrated probability.

3. 3

   Draw the serving path

   Candidate filter, feature lookup, model, calibrator, auction, impression log. Give each step a slice of the time budget.

4. 4

   Draw the training path

   Join views and clicks with a wait window, downsample non-clicks, correct for it, train often, and gate releases.

5. 5

   Pick a model and defend it

   Start with logistic regression on hashed features and crosses. Name a deeper model as the next step, behind an experiment.

6. 6

   Cover metrics and risks

   Log loss, AUC and calibration per group offline. Revenue and CTR online. Mention exploration for new ads and a fallback model.