Two-Tower Models: Retrieval Then Ranking

It runs out of time. A careful model that judges one user and one item together is too slow to run on a whole catalogue while a page loads. You need a cheap first pass that picks a shortlist, and a two tower model is that pass.

Picture a common setup. An online marketplace in Kochi sells handicrafts from about 9,000 small artisans: coir doormats, brass lamps, bamboo baskets and handloom sarees. It lists around 200,000 items. The setup is made up, but the problem is real.

The team has a ranking model for the home page. It reads 120 signals about one shopper and one item together, then predicts how likely that shopper is to buy. It takes about 0.1 milliseconds (ms) for each shopper and item pair.

The naive plan is to run that ranker on everything. Scoring all 200,000 items takes 200,000 times 0.1 ms, which is 20 seconds. The page has about 150 ms in total, so the naive plan misses by more than 100 times.

So the team needs a much faster model in front. It must pick around 500 items worth the ranker's time, in a few milliseconds. Scoring 500 items then takes the ranker only 50 ms.

A two tower model is a neural network with two separate halves. One turns a user into a list of numbers, and the other turns an item into a list of the same length. The match score is the dot product of the two lists.

That sentence uses three words worth slowing down for. Here they are, with examples from the marketplace.

Embedding. An embedding is a fixed-length list of numbers that stands for something, such as a shopper or an item. The marketplace uses 64 numbers. Training arranges them so a shopper and the items they like score high together.

Tower. A tower is a small neural network, a stack of layers that turns input features into an embedding. The user tower reads things like recent clicks, past searches and city. The item tower reads the title, category, price band, artisan's region and photo features.

Dot product. A dot product multiplies two lists position by position and adds the results. A higher number means a better match.

Here is a tiny worked example with 3 numbers instead of 64. Say one shopper's embedding is 0.9, 0.1 and 0.4. A coir doormat is 0.8, 0.0 and 0.5, so the score is 0.72 plus 0 plus 0.20, which is 0.92. A brass diya is 0.1, 0.9 and 0.2, which scores only 0.26.

What just happened? The shopper and the doormat have their big numbers in the same positions, so the products add up. The diya's big number sits where the shopper's is small, so it scores low.

The key point is the shape of the model. Follow the two columns in the diagram below and notice that they meet only at the very last box.

How does it learn good numbers? From pairs that really happened, such as a shopper adding a basket to the cart. Training pulls each real pair's scores up and pushes other pairs down. The details, and a trap hidden in them, come in the engineer half.

The natural language community calls the same design a dual encoder, as a 2020 recommender paper notes. An encoder is anything that turns an input into an embedding. In search, one tower embeds the query and the other embeds the documents.

It is fast because every item embedding is computed before any shopper arrives. At request time, the system runs only the user tower once. Then it looks up the item embeddings that score highest against that one shopper embedding.

A city map helps here. Map each part of the picture to the system:

• Each item's embedding is a shop's pin on the map, placed once, before anyone visits.
• The shopper's embedding is your own pin, dropped when you open the app.
• Retrieval is finding the 500 pins that match yours best.
• An ANN index is the map's grid of neighbourhoods, so you search a few squares instead of every pin.
• Ranking is then visiting those 500 shops to check each one in person.

Where the picture breaks: a map has 2 directions and these embeddings have 64. A dot product is not distance either: an item with big numbers overall scores high with many shoppers, like a shop with an oversized sign. And every retrain redraws the whole map.

The marketplace's item tower runs once a night over all 200,000 listings. A shopper's visit then needs one pass through the user tower and one lookup. With 200,000 items, the lookup can simply check every item, which is called brute force.

With tens of millions of items, checking every one per visit gets too slow and too costly. Then you use an ANN index. It finds items that are very close without checking them all, at the price of sometimes missing a few true best ones.

It needs a ranking stage because the towers never see the shopper and the item together, so they miss signals that only exist for the pair. A slower ranking model, run on just the shortlist, can use those signals.

Back to the map: the pins found the 500 nearby shops, but only a visit tells you if a shop is open today. In the same way, the towers cannot see pair signals like these:

• Whether this shopper already bought this exact item last week.
• How the item's price compares with what this shopper usually spends.
• Whether the artisan ships to this shopper's pin code in under 5 days.
• How often this shopper clicked this item's category in the last hour.

The team's ranker reads all 120 signals for each of the 500 pairs, including these. It also applies business rules the towers should not learn, such as hiding out-of-stock items and not showing four lamps in a row.

Real systems also feed the ranker from more than one place. The marketplace mixes two-tower picks with new arrivals, items popular in the shopper's city and more from artisans they bought from before. The ranker sorts all of them together.

This split has a long published record. A 2016 paper on video recommendations described a candidate generation network that pulls hundreds of videos from millions. A separate ranking network, with many more features, then orders them.

One detail an expert will spot: that 2016 candidate generator was not a full two tower model. It learned one embedding for each video ID directly, like a lookup table, instead of computing it from item features. A 2019 paper from the same video site used a true item tower.

Retrieval picks a few hundred items from the whole catalogue and cares most about not missing good ones. Ranking orders that shortlist and cares most about getting the top few right. They need different models because their jobs differ.

Two terms help here. Recall means the share of items a shopper would like that made it into the shortlist. Precision at the top means the share of the first few shown items that are actually good.

A common beginner question is why not put the ranker's model inside the towers. You can make the towers bigger, but they still cannot mix shopper and item inputs. The moment they do, item embeddings depend on the shopper and cannot be computed ahead of time.

You score every item with one matrix multiplication and keep the best k. The code below is the simplest honest version. The helper user_tower stands in for the trained user tower and returns 64 numbers for one shopper. The two .npy files come from the nightly item job, so to run something now, use the check further down.

import numpy as np

item_ids = np.load("item_ids_v12.npy")    # 200,000 listing ids
item_vecs = np.load("item_vecs_v12.npy")  # shape (200000, 64), built overnight

def candidates(shopper, k=500):
    u = user_tower(shopper)                 # one pass, 64 numbers
    scores = item_vecs @ u                  # dot product with every listing
    top = np.argpartition(-scores, k)[:k]   # best k, in no particular order
    best_first = top[np.argsort(-scores[top])]
    return item_ids[best_first]

Two lines matter. The @ line scores all 200,000 listings in one matrix multiplication, about 12.8 million multiply-and-add steps. The argpartition line picks the best 500 without sorting all 200,000 scores.

We ran this function with made-up vectors on a 2-core test machine. It took about 7 ms per call, the median of 200 runs. For 100 test shoppers it returned the same 500 items as a full sort, in score order.

Items with exactly equal scores can swap places, which happened for 4 of the 100. If your logs or tests need a repeatable order, break ties by item id.

Notice the v12 in both file names. The ids and the vectors must come from the same model version. That detail comes back in the serving section, because mixing versions quietly breaks retrieval.

Check it on your own machine

This script times the same two lines on random vectors. It needs only NumPy, the usual Python library for arrays of numbers (pip install numpy).

import time
import numpy as np

rng = np.random.default_rng(0)
item_vecs = rng.standard_normal((200_000, 64), dtype=np.float32)
u = rng.standard_normal(64, dtype=np.float32)

times = []
for _ in range(20):
    start = time.perf_counter()
    scores = item_vecs @ u
    top = np.argpartition(-scores, 500)[:500]
    times.append((time.perf_counter() - start) * 1000)

print(item_vecs.nbytes / 1e6, "MB of item vectors")
print(len(top), "candidates, median", round(float(np.median(times)), 1), "ms")
print("same as a full sort:", set(top) == set(np.argsort(-scores)[:500]))

The first line should read 51.2 MB of item vectors: 200,000 items times 64 numbers times 4 bytes each. Tools that count in 1,024s show that as 48.8 MiB. The last line should read True.

The middle line printed a median of 6.8 ms on our 2-core test machine. Your number will differ with your CPU, but it should stay in milliseconds, not seconds.

When brute force stops being enough

One popular ANN index is HNSW, short for hierarchical navigable small world graphs. Items are linked to their near neighbours in several layers. A search starts on a sparse top layer and greedily moves to closer items as it goes down.

The 2016 paper that introduced HNSW shows search effort growing roughly with the logarithm of the item count. Its proof assumes an idealised graph, and its simulations that confirm it use low-dimensional data. So treat the scaling as a strong tendency, not a promise.

A search-width setting, called ef in the paper, controls how many candidates the search keeps. A wider search finds more of the true best items and takes longer.

Measure what the index misses

We built an HNSW index over 200,000 made-up, clustered vectors of 64 numbers with an open-source HNSW library. Then we compared its top 500 with brute force for 1,000 test shoppers.

• Search width 500: it found 93.9% of the exact top 500.
• Search width 800: 97.2%.
• Search width 1,500: 99.3%, with about three to four times the search time of width 500 in our runs.

Your data will give different numbers, but the shape of the trade holds. Run a few thousand past shoppers through both brute force and your index, and pick the width your latency budget allows.

A two-tower model is trained on pairs of users and the items they engaged with. It learns to score each true pair higher than pairs that did not happen. The tricky part is choosing those non-pairs, called negatives.

Positive. A positive is a pair that really happened, such as a shopper who added a bamboo basket to the cart. The marketplace logs about 400,000 clicks and 30,000 cart adds a day. The team uses cart adds and purchases, since clicks alone reward pretty photos.

Negative. A negative is a pair you treat as not wanted. You rarely know what a shopper truly disliked, because most items were simply never shown. So negatives are a guess, and the guess shapes what the model learns.

The common trick is in-batch negatives. Training reads pairs in groups called batches, say 1,024 pairs at once. For each shopper in the batch, its own item is the positive. The other 1,023 items in the same batch act as its negatives, for free.

The function below computes the loss, the number training tries to make smaller. Row i of user_vecs and row i of item_vecs form one real pair. Both are assumed to be scaled to length 1, which keeps each dot product between minus 1 and 1.

import numpy as np

def in_batch_loss(user_vecs, item_vecs, temperature=0.05):
    # row i of each matrix is one real (shopper, item) pair
    scores = user_vecs @ item_vecs.T / temperature    # every shopper vs every item
    scores -= scores.max(axis=1, keepdims=True)         # avoids overflow in exp
    log_probs = scores - np.log(np.exp(scores).sum(axis=1, keepdims=True))
    return -np.mean(np.diag(log_probs))                 # true pairs sit on the diagonal

The first score line builds a 1,024 by 1,024 grid: every shopper against every item in the batch. The log_probs line turns each row into probabilities with a softmax, a formula that makes a row of scores add up to 1.

The last line reads the diagonal, where each shopper meets its own item. Training pushes those probabilities up, which pushes the other items in each row down. Dividing by the temperature, a small number like 0.05, stretches the gaps between scores.

The max subtraction does not change the result, because softmax ignores a shift. It stops exp from overflowing when scores are large. In our test, 32-bit numbers at temperature 0.01 overflowed without it and stayed finite with it.

Check the loss before you train

Pause and predict. Pure guessing among 1,024 items gives a loss of ln(1024), about 6.93. What will random, untrained towers score at temperature 0.05? Run this to find out.

import numpy as np
# paste in_batch_loss from the block above first

rng = np.random.default_rng(0)

def unit(x):
    return x / np.linalg.norm(x, axis=1, keepdims=True)

users = unit(rng.standard_normal((1024, 64)))
items = unit(rng.standard_normal((1024, 64)))

print("random towers:", round(in_batch_loss(users, items), 2))
print("pure guessing:", round(float(np.log(1024)), 2))
print("perfect towers:", round(in_batch_loss(users, users), 2))

On NumPy 2.1.3 it prints 9.96 for random towers, 6.93 for pure guessing and 0.0 for perfect towers. The perfect case uses each shopper's own vector as its item, so every true pair wins easily.

What just happened? Random towers scored worse than guessing. The low temperature stretches random scores, so the softmax becomes confidently wrong. At temperature 1, the same random towers sit at 6.94 in our runs.

So judge a model's first loss against its temperature, not against 6.93 alone. A loss that never drops below the random-tower level means the towers are not learning.

The popularity trap in in-batch negatives

In-batch negatives have a hidden bias. Say the best-selling coir doormat is 1 in every 100 cart adds. A batch of 1,024 pairs then holds about 10 doormats, and each one is a negative for nearly every other shopper in it.

A temple lamp bought once in 200,000 cart adds turns up in only about 1 batch in 196. In our simulation of these made-up rates, the doormat served as a negative about 2,000 times as often as the lamp.

A 2019 paper on two-tower retrieval for a large video site named this sampling bias. Popular items are over-punished as negatives simply because they are so likely to be in a batch.

Its fix estimates p, the chance that an item lands in a batch, from the stream of training data. It then subtracts the log of p from that item's temperature-scaled score in every row, before the softmax. The paper applies this inside the training loss.

Here is what that does to our two items. The doormat sits in almost every batch, so p is near 1 and its score barely changes. The lamp's p is about 0.0051, so its score gets about 5.3 added. Each lamp appearance now pushes harder than a doormat appearance, which shrinks the gap between them.

A 2020 paper on app-store recommendations attacked the opposite gap. An item nobody interacted with never appears in a training pair, so in-batch sampling can never pick it as a negative. Long-tail items rarely get picked either.

Its mixed negative sampling adds items sampled uniformly from the whole catalogue to each batch's negatives. Its best reported setting added 8,192 such items to batches of 1,024 pairs.

You serve it in two halves. An offline job turns every item into an embedding and builds the index, and an online service runs the user tower per request. Both halves must always use the same model version.

Here is the offline half in the marketplace setup. Follow it from the training job down to the index, and notice that every output carries the version number.

Why must the versions match? Each training run arranges its 64 numbers in its own way, and nothing ties position 7 in v12 to position 7 in v13. A v13 shopper against v12 items gives confident nonsense.

We tested this by modelling a retrain as a random rotation of the space, which keeps quality but changes the axes. Over 200 shoppers, a rotated shopper against unrotated items kept on average under 1 of the true top 500. Random picks would keep about 1.25.

New listings need care too. An artisan who uploads a new mirror at 10 a.m. should not wait for tonight's job. So the team runs the item tower on each new or edited listing within minutes and adds it to the live index.

This helps because the item tower reads content, such as the title, category and photo. A tower that relied only on item IDs would have nothing learned yet for a brand-new ID. Content gives a new item a sensible place before anyone clicks it.

That is a big help with the cold start problem, where new items have no history. For a real reference point, the 2019 video paper's system received new training data every day and rebuilt its index every few hours.

Now the online half. Each box below is one step of a single home page request, in order. The second line on each box says which parts talk.

The user tower runs on every visit, because a shopper's last few clicks matter. If a shopper just looked at three brass lamps, their embedding should already lean toward lamps on the next page.

The failures below come from the data and the plumbing, not the towers themselves. The common ones are biased negatives, mismatched versions, stale indexes and a shortlist that slowly narrows to the same popular items.

• Version mismatch. The user tower moves to v13 while the index still holds v12. Guard: one version tag on everything, and refuse to serve mixed versions.
• Popularity bias. Best sellers crowd out small artisans. Guard: correct for item frequency in training and track how many artisans appear in shortlists.
• Stale index. A sold-out saree keeps showing up. Guard: filter stock after retrieval, and rebuild or patch the index on time.
• Feedback loop. The model learns only from what it showed, so it shows the same things again. Guard: mix in other sources and some fresh items.
• Silent recall drop. Ranking looks fine, but good items never reach it. Guard: track recall at 500 on a fixed set of past shoppers every day.

The feedback loop is easy to miss. Each week the training data holds more of what the last model chose. Without fresh sources, the part of the catalogue a shopper can reach quietly shrinks.

Explain it as the fast first stage of a two-stage recommender. Then justify every choice with a number from the catalogue size and the time budget.

1. 1

   Do the scoring maths

   Multiply catalogue size by the ranker's time per item. Show that scoring everything misses the page budget by a wide margin.

2. 2

   Split retrieval and ranking

   Say retrieval picks a few hundred items with high recall, and ranking orders them with a richer model over pair features.

3. 3

   Draw the two towers

   A user tower and an item tower, each producing an embedding, joined only by a dot product.

4. 4

   Move item work offline

   Embed all items in a batch job, build an ANN index, and run only the user tower per request.

5. 5

   Explain the training

   Positives from purchases or long views, in-batch negatives, and a correction for popular items appearing too often.

6. 6

   Cover freshness and versions

   Embed new items within minutes, tag every artefact with one model version, and swap the index only after a recall check.