Recommendation System Design: Candidate Generation and Ranking

When everyone gets the same home screen, most people see rows picked for someone else, and many leave without playing anything. Recommendation system design fixes this by choosing a different short list for each person.

Picture a common setup. A Telugu video-streaming app has about 50,000 titles: films, web series episodes, comedy clips and devotional songs. About 8 lakh (800,000) people open it on a normal day. The setup is made up, but the problem is real.

An editor team builds the home screen by hand, so everyone sees the same rows: Trending, New this week and Top films. A grandmother who only watches devotional songs sees the same action films as a college student.

The product team reads the numbers every Monday. Of every 100 people who open the app, 42 leave without playing anything. The team wants a home screen built for each person, so more people find something to watch in their first minute.

The naive fix is one careful model that scores every title for every visit. At the Sunday 9 pm peak of about 150 visits a second, that is 75 lakh (7.5 million) scores a second. Almost all of that work goes on titles that never had a chance.

Recommendation system design is planning the pipeline that picks the best few items for one person from a huge catalogue, fast enough that the screen feels instant. An item is whatever you recommend. Here, it is a video.

The hard part is scale. Published large-scale recommenders split the work into stages, each narrower and more careful than the one before. Early stages spend almost nothing per item, so later stages can spend a lot.

Picture a crowded bookstall on College Street in Kolkata, with thousands of books stacked to the ceiling. A regular asks for something to read on the train. In a few seconds the bookseller pulls a rough pile of twenty, without weighing every book.

Then he looks closely at those twenty. She finished a long novel last week and likes thin books for her commute. He hands her three, surest pick on top, and never three by the same author.

• The rough pile is candidate generation: cheap, quick, and fine if a few poor books sneak in.
• The close look is ranking: slow and careful, but only over twenty books.
• Never three by one author is re-ranking: a final pass for variety.
• His memory of what she bought is the training data the next model learns from.

Where the picture breaks: the bookseller can simply ask her what she wants. A recommender mostly guesses from what people did, and it must do so for about 150 people a second.

Three words come up again and again in this guide:

• Candidate: a video that survives the first, rough stage and gets a careful score later. The app keeps about 1,000 per request.
• Feature: one fact the model uses, written as a number or a category. Example: the share of this person's watching that is in Telugu.
• Label: the outcome the model learns to predict from past data. Example: did this person watch at least 2 minutes after tapping a video?

Here is the whole pipeline. Read it as a funnel, then follow the arrows from the event log, which turn today's watching into tomorrow's training data.

Candidate generation runs several cheap sources side by side, each returning a few hundred videos this person might like, then merges them into one list. It aims for recall, meaning every good video should be somewhere in the list, even if the order is rough.

Recall matters because the ranker can only score what it is given. A film that never becomes a candidate cannot be rescued later.

The sources, from simple to learned

• Continue watching: unfinished titles and the next episode of a series in progress. Read live, because they change minute by minute.
• Also watched: people who finished this film often watched these next. A nightly job counts these pairs, a simple form of collaborative filtering (learning from what similar people did).
• Trending: the most-played videos in the last 24 hours, per language. Cheap, shared by many users, and useful for people with little history.
• New releases: titles added this week, so fresh videos get a chance before anyone has watched them.
• Two-tower search: a learned model that finds videos close to this person's taste, explained just below.

Each source returns a fixed share, called a quota, so one loud source cannot crowd out the rest. A merge step then removes repeats and drops videos this profile must never see, such as adult titles on a kids profile.

The two-tower model, in brief

A two-tower model is a neural network (layers of simple maths learned from data) with two halves that meet only at the very end. The user tower turns a person's history into an embedding, a list of numbers that places similar things close together. The video tower does the same for each video.

Training pulls a person's embedding close to the videos they really watched and away from others. Video embeddings do not depend on the person, so all 50,000 are computed ahead of time. At request time the app computes one user embedding and looks up the nearest videos.

That lookup uses approximate nearest neighbour search, or ANN: finding very close embeddings without checking every one, at the cost of sometimes missing a true neighbour. HNSW, a graph-based index from a 2016 paper, is one such method.

Its authors report search cost that grows roughly with the logarithm of the number of items. They derive it for idealised graphs, and their scaling evidence is mainly on low-dimensional data. On 200 million image-feature vectors, they saw search cost drift from a pure logarithm, so treat it as a guide.

The price of that speed is a blind spot: the towers meet only at one final similarity score. Neither tower can take a fact that needs this person and this video together as an input, such as having finished 3 of this director's 4 films. That is the ranker's job. Our two-tower model guide covers training and serving in depth.

What about new users and new videos?

This is the cold start problem: a new user or a new video has no history, so models that learn from history have nothing to go on. The app handles it in a few plain ways, and our cold start problem guide goes deeper.

• New user: ask for a language and three favourite stars at sign-up. Lean on trending and new-release sources until a few plays come in.
• New video: describe it with content features such as language, genre, cast and length. If the video tower takes those as inputs, it can place a new video on day one.
• Both: keep a few exploration slots, where fresh videos are shown to a small share of users so the system can learn about them.

The ranking stage runs a heavier model on each of the roughly 1,000 candidates, predicting what this person will do with each video. It then sorts them by one score built from those predictions. Here the goal is precision: the top 10 must be right.

The ranker can afford far more features than the towers, because it scores a thousand videos, not fifty thousand. The team starts with a gradient-boosted tree model: many small decision trees added one at a time. Each new tree is fitted to the errors the earlier trees still make (strictly, to the negative gradient of the loss).

The team picks trees for a first ranker because they train quickly and are easy to inspect. Features fall into a few groups, shown here with examples from the app.

Features must be computed the same way in training and in serving. Say training measured Telugu share over 30 days, but serving uses 7 days. The model then sees numbers unlike the ones it learned from.

That mismatch is one cause of training-serving skew: a gap between how a model does in training and how it does live. A well-known set of ML engineering rules gives two habits against it. Reuse the same feature code in both places, and log the features used at serving time to train on later.

A feature store, one shared place that computes, stores and serves features, is a common way to do both. Our feature store guide explains how one works.

Usually neither on its own. Clicks reward eye-catching thumbnails, and raw watch time rewards long films, so teams often predict several outcomes and blend them into one score.

Click-through rate is the share of times a video is shown that it gets a tap. A 2016 paper on a large video platform's recommender says ranking by it often promotes deceptive videos that people do not finish. Its ranker predicts expected watch time instead.

The team in our setup learns this the hard way. Its first ranker predicts only taps on a video tile. In a small test, taps rise by 12% but weekly watch minutes fall by 4% (made up).

The logs show why. Videos with shocking thumbnails win the taps, then people quit within 30 seconds. The model did exactly what it was told, and it was told the wrong thing.

Pure watch time has its own trap. A three-hour film beats a 20-minute episode on minutes, even when the person would enjoy the episode more. So the team's ranker predicts four things for each candidate:

• Play: will they tap and watch at least 2 minutes?
• Minutes: how long will they watch if they play?
• Finish: will they reach the end, or start the next episode?
• Not interested: will they tap the not-interested button, or skip within seconds?

A final score blends the four with weights. The weights are product decisions, so the product team and the ML engineers set them together and test them online.

Walk each source in order, take items until its quota is full, and skip repeats, finished titles and anything this profile may not see. This short version runs on plain Python 3 with no libraries.

def gather_candidates(sources, finished, allowed, limit=1000):
    seen = set(finished)              # never suggest what they finished
    merged = []
    for name, quota, ids in sources:  # each source lists its best first
        taken = 0
        for vid in ids:
            if taken == quota:
                break
            if vid in seen or not allowed(vid):
                continue              # repeat, finished or not allowed
            seen.add(vid)
            merged.append((vid, name))
            taken += 1
    return merged[:limit]

sources = [
    ("continue", 2, ["ep5", "film9"]),
    ("two_tower", 3, ["film1", "film2", "film3", "film4"]),
    ("also_watched", 2, ["film2", "film7", "film8"]),
    ("trending", 2, ["film1", "adult1", "film5"]),
]
kids_ok = lambda vid: not vid.startswith("adult")
for vid, name in gather_candidates(sources, {"film3"}, kids_ok):
    print(name, vid)

Run it with python3 and you should see these eight lines:

continue ep5
continue film9
two_tower film1
two_tower film2
two_tower film4
also_watched film7
also_watched film8
trending film5

What just happened? film3 is skipped because the person finished it, and film2 appears once although two sources returned it. adult1 is blocked by the kids filter, so trending adds only film5 because its short list ran out.

The quota counts only items a source actually adds, so each source gets its full share of new videos. The order of the merged list does not matter, because the ranker sorts it again. In production, the app's quotas are 50 for continue watching, 400 for two-tower search, 300 for also watched, 150 for trending and 100 for new releases, 1,000 in all.

Re-ranking is a light final pass that reshapes the ranker's sorted list. It keeps rows from repeating themselves, gives fresh titles a fair chance, and enforces business rules.

Without it, a fan of one series might get ten episodes of that series in one row. Each scores high alone, but together they make a dull row.

A classic method is maximal marginal relevance, or MMR, from 1998. It picks each next item for being relevant and unlike the items already picked, with one weight setting the balance. The team starts with something simpler: a cap per series and one slot for something new.

def final_score(p):
    return p["play"] * p["minutes"] + 5 * p["finish"] - 50 * p["not_interested"]

def build_row(ranked, size=10, max_per_series=2):
    row, per_series = [], {}
    for v in ranked:                          # highest score first
        if per_series.get(v["series"], 0) >= max_per_series:
            continue                          # too many from this series
        row.append(v)
        per_series[v["series"]] = per_series.get(v["series"], 0) + 1
        if len(row) == size:
            break
    if not any(v["days_old"] <= 7 for v in row):
        fresh = [v for v in ranked if v["days_old"] <= 7 and v not in row
                 and per_series.get(v["series"], 0) < max_per_series]
        if fresh:
            row[-1] = fresh[0]                # keep one new title
    return row

preds = {
    "clickbait clip": {"play": 0.6, "minutes": 2, "finish": 0.1, "not_interested": 0.3},
    "drama episode": {"play": 0.3, "minutes": 22, "finish": 0.6, "not_interested": 0.02},
    "3-hour film": {"play": 0.2, "minutes": 95, "finish": 0.3, "not_interested": 0.05},
}
for name, p in preds.items():
    print(name, round(final_score(p), 1))

ranked = [
    {"id": "S ep1", "series": "S", "days_old": 300},
    {"id": "S ep2", "series": "S", "days_old": 300},
    {"id": "S ep3", "series": "S", "days_old": 300},
    {"id": "film A", "series": "A", "days_old": 90},
    {"id": "film B", "series": "B", "days_old": 40},
    {"id": "film N", "series": "N", "days_old": 2},
]
print([v["id"] for v in build_row(ranked, size=4)])

Expected output:

clickbait clip -13.3
drama episode 8.6
3-hour film 18.0
['S ep1', 'S ep2', 'film A', 'film N']

What just happened? final_score turns four predictions into one number. Play chance times expected minutes gives expected minutes, finishing adds a bonus, and a likely not-interested tap costs a lot. The clickbait clip sinks to -13.3.

Notice the three-hour film still beats the episode, 18.0 to 8.6. Raw minutes favour long titles, and this blend only softens that. Capping the minutes term, or predicting the share of the title watched, are options to test online.

build_row walks down the sorted list and allows at most two videos per series, with each film as its own series. S ep3 is dropped by the cap. The row had nothing from the last 7 days, so film N takes the last slot from film B.

Some rules are not about taste at all, and they always win:

• Kids profiles never see titles above their age rating.
• Videos a person marked not interested stay hidden for 90 days.
• A promoted film may get one fixed slot, chosen by the product team rather than the model.

Position bias means videos near the top get more taps simply because more people look there. Raw click rates then mix up good with well placed, and a model trained on them learns the old layout as well as taste.

A 2016 paper on unbiased learning-to-rank (training a model to order results) says using click data directly as a training signal gives sub-optimal rankings. Its fix weights each click by one over the chance that its position was looked at, so clicks in rarely seen slots count more.

Here is a small simulation. Video A is truly more appealing than video B, but the old system put B in the top slot 80% of the time. All the numbers are made up.

import random

random.seed(7)
look = {1: 1.0, 5: 0.25}           # chance a viewer looks at the slot (made up)
appeal = {"A": 0.30, "B": 0.20}    # chance of a tap if they look (made up)
shows, taps, weighted = {"A": 0, "B": 0}, {"A": 0, "B": 0}, {"A": 0.0, "B": 0.0}

for _ in range(100_000):
    top = "B" if random.random() < 0.8 else "A"   # old system favours B
    other = "A" if top == "B" else "B"
    for video, slot in ((top, 1), (other, 5)):
        shows[video] += 1
        if random.random() < look[slot] and random.random() < appeal[video]:
            taps[video] += 1
            weighted[video] += 1 / look[slot]     # rarely seen slot, bigger weight

for v in "AB":
    print(v, "raw", round(taps[v] / shows[v], 3),
          "corrected", round(weighted[v] / shows[v], 3))

Expected output (we ran Python 3.13.16):

A raw 0.121 corrected 0.301
B raw 0.169 corrected 0.199

What just happened? B's raw rate of 0.169 beats A's 0.121, because B mostly sat on top. Dividing each tap by the chance its slot was looked at recovers the true appeal: about 0.30 for A and 0.20 for B.

We ran it with 20 different seeds, and B won on raw rate every time. The corrected rate for A stayed between 0.293 and 0.308.

The catch is that real look chances are unknown. The paper estimates them with a small experiment that swaps results between positions. It found that even imperfect estimates can give a big improvement over ignoring the bias, but underestimating small chances can hurt.

A simpler fix feeds the position into the model as a feature during training, then uses one fixed default position for every candidate at serving time. A 2021 paper calls this widely adopted in industry and names its weakness: training and serving treat position differently.

Fast enough that the home screen feels instant. The team's target is 250 ms at p99 on the server, meaning 99 of every 100 requests finish within 250 milliseconds.

Scale makes this hard. At the Sunday 9 pm peak, the app serves about 150 home screens a second, so the ranker scores 1.5 lakh (150,000) user-video pairs every second.

The team splits the budget across the steps, with made-up timings:

• User features from the feature store: 20 ms.
• Candidate sources, run side by side, then merged: 40 ms.
• Features for the 1,000 candidates, fetched in batches: 40 ms.
• Ranker scoring: 80 ms.
• Re-ranking and building rows: 10 ms.
• Spare for network delays: 60 ms.

Three habits keep the budget. Precompute what you can: also-watched pairs and trending lists are built every night or hour, then cached. Batch the work: the ranker scores all 1,000 candidates in one call, not 1,000 calls.

And always keep a fallback. If ranking runs late, the app gets cached trending rows for the person's language. A slightly worse home screen beats a spinning one.

To check the budget, record each step's time on every request and track the p99 of each step, not only the total. A step that creeps from 40 ms to 70 ms has eaten half the 60 ms spare while the total still looks fine.

First offline, on past data, to catch bad models cheaply. Then online, with an A/B test on real users, to measure what actually matters. An A/B test splits users at random between the old system and the new one.

Offline, the team trains on data up to 30 September and tests on the first week of October. A random split lets the model peek at the future, which makes it look better than it will be live.

• Recall at 1,000, for candidate generation: of the videos each person really played in the test week, what share were among their candidates?
• NDCG at 10 (normalised discounted cumulative gain), for ranking: a score from 0 to 1. Each played video in the top 10 earns 1 / log2(position + 1), and the total is divided by the best possible total.
• Coverage: the share of all 50,000 titles recommended to anyone at all. Low coverage warns of a narrow, repetitive system.

This code computes recall and NDCG for one person, both at 10 on the ranked list to keep the example small. In a real test you average them over everyone in the test week.

import math

def recall_at_k(ranked, played, k):
    return len(set(ranked[:k]) & played) / len(played)

def ndcg_at_k(ranked, played, k=10):
    dcg = sum(1 / math.log2(i + 2) for i, v in enumerate(ranked[:k]) if v in played)
    best = sum(1 / math.log2(i + 2) for i in range(min(len(played), k)))
    return dcg / best

played = {"film2", "ep5", "film8"}      # played in the test week
ranked = ["film1", "film2", "film4", "ep5", "film7",
          "film9", "film5", "film6", "film3", "film10"]
print(round(recall_at_k(ranked, played, 10), 3))
print(round(ndcg_at_k(ranked, played), 3))

Expected output:

0.667
0.498

What just happened? Two of the three played videos made the top 10, so recall at 10 is 0.667. They sit at positions 2 and 4 rather than 1 and 2, so NDCG at 10 is 0.498. Our numbers match scikit-learn 1.5.2's ndcg_score on the same data.

Offline results can still mislead, because past data only shows reactions to what the old system chose to show. The 2016 video recommender paper notes that live A/B results are not always correlated with offline experiments.

Re-weighting old logs to estimate how a new system would have done has limits too. A 2018 study checked such estimators against 39 A/B tests on a large commercial recommender. Two standard ones gave poor bias-variance trade-offs (the balance between being wrong on average and being noisy) for personalised product recommendation.

Here is how the team runs its A/B test:

1. 1

   Pick one main metric

   Weekly watch minutes per user, agreed with the product team before the test starts.

2. 2

   Pick guardrails

   Not-interested taps, home screen load time, app crashes and kids-profile complaints must not get worse.

3. 3

   Run the test

   A random 5% of users get the new system and another 5% keep the old one, for two weeks. Two weeks covers two weekends.

4. 4

   Read it honestly

   Check the main metric, then every guardrail, then key groups such as new users and TV viewers.

5. 5

   Roll out in steps

   Go to 25%, then 50%, then everyone, watching the same numbers at each step.

A feedback loop happens because the system learns from what it showed. People can mostly watch only what was recommended, so tomorrow's training data repeats today's choices. Over time this can narrow what each person sees and what the whole app promotes.

Back on College Street, suppose the bookseller only ever suggests detective novels. She buys them, so he remembers that she loves detective novels. He never learns she loves poetry, because he never puts a poetry book in her hands.

A 2019 paper on degenerate feedback loops separates two effects. An echo chamber is a person's interest being reinforced by repeated exposure. A filter bubble is the system serving only limited content.

It calls a system degenerate when a person's interest keeps drifting further from where it started. In its theory and simulations on synthetic data, the best remedies were continuous random exploration and growing the candidate pool at least linearly. It also found that adding random exploration to a model that already knows people's true interests can speed that drift up.

In the app, popular films get more slots, so they get more plays, so they look even better to the model. Small Telugu indie films never get shown, so they never get data.

• Exploration: fill about 5 of every 100 slots with videos the model is unsure about, and learn from the reaction.
• Keep adding new titles to the candidate pool, through the new-releases source.
• Log impressions, meaning every video shown and its position, not just plays.
• Keep a small holdout group on plain trending rows, for long-term comparison.
• Track coverage and the share of plays going to the top 100 titles, every week.

Walk the funnel from goal to feedback loop, with a number on every stage and a reason for every choice. For the general shape of these rounds, see our guide on the ML system design interview framework.

1. 1

   Clarify the goal

   Ask what the business wants, then agree on one main metric and a few guardrails.

2. 2

   Size the problem

   Catalogue size, daily users, peak requests per second and the latency budget.

3. 3

   Draw the funnel

   Candidate sources, ranking and re-ranking, with a rough count at each step: 50,000, 1,000, 100, 60 shown.

4. 4

   Choose labels and features

   Explain why clicks alone are risky, what you predict instead, and the main feature groups.

5. 5

   Cover serving

   Precomputed sources, batched scoring, a feature store, and a fallback when ranking runs late.

6. 6

   Evaluate and close the loop

   Time-split offline tests, position bias, an A/B test with guardrails, cold start, exploration and impression logging.