Design Search Ranking for a Rental App

Renters see unproven listings first and book less, because a listing's age says almost nothing about whether a renter wants it. Fixing that is the job of search ranking system design: deciding which results go on top, and proving the order works.

Picture a common setup. A rental app for flats and PG rooms in Bengaluru has about 22,000 listings. A PG, short for paying guest, is a bed or room in a shared home, rented monthly, often with meals. The setup is made up, but the problem is real.

Renters search by area, budget and move-in date. To hold a place, a renter pays a token amount of Rs 2,000, and the app calls that a booking. Results are sorted newest first, and about 35,000 searches a day end in only 280 bookings.

The search team reads a week of search logs and finds four kinds of trouble. Each one gets its own fix later in this guide.

• Fresh listings with one blurry photo and no reviews fill the top of a Koramangala search under Rs 15,000 a month. A ranking model fixes this.
• A well-reviewed PG in HSR Layout sits on page three, only because it was listed a year ago. Position bias, covered later, explains why the logs hide how good it is.
• Some owners delete and re-post the same flat every morning, just to jump back to the top. A duplicate rule ends this.
• Flats already rented out still show up, so renters call owners who say the room is gone. A hard filter ends this.

The team's goal is easy to say and hard to build: put the listings this renter is most likely to book at the top. The rest of this guide builds that, one piece at a time.

You need six words: query, candidate, feature, label, ranking model and impression. Each one is explained here with the rental app's own example.

Query. The query is what the renter asked for, including filters. Example: a PG in Whitefield, under Rs 9,000 a month, for a woman, moving in on 1 December.

Candidate. A candidate is a listing that passes the basic filters and might be shown. A Whitefield search may have 600 candidates.

Feature. A feature is one number or fact the model looks at, such as rent, distance to the renter's office, or how fast the owner replies.

Label. A label is the past outcome a model learns to predict. Here the main label is simple: did the renter book this listing or not?

Ranking model. A ranking model is a program, trained on past data, that gives each candidate a score. Higher scores go higher on the page. In this app, the score means: how likely is this renter to book this listing?

Impression. An impression is one listing shown to one renter in one search, at one position. Every impression is logged, whether or not the renter clicked it.

Search ranking system design is planning the system that decides the order of search results: what to fetch, how to score it, and how to check the order is good. Many large sites score results with a model trained on what past users chose.

It is more than a sort function. It covers the search index, the stored features, the model and the rules after it. It also covers the logs that train the next model, and the tests that decide whether a change ships.

You rank by booking likelihood because a booking is the outcome the renter and the business both want, while a click only shows curiosity. A model trained on clicks learns to promote listings that look good, not ones that get rented.

Think of a fishmonger at a morning bazaar in Kolkata. She has one bed of crushed ice at the front of her stall. She does not put the fish that arrived last on top. She puts the fish most likely to sell to this morning's buyers.

Her ordering is a prediction about her buyers, and so is a ranking. The ice bed is the first page, the fish are listings and the buyers are renters. She also does not care which fish people touch and put back. She cares which fish get weighed and paid for.

Clicks are the touching. Bookings are the paying. Where the picture breaks: she watches every buyer's face and changes her display within minutes. A ranking model knows only what was logged, and its learned weights change only when you retrain it.

The team saw this in its own logs. One Indiranagar flat had wide-angle photos and the most clicks in its area. It asked Rs 6,000 a month more than similar flats, and nobody booked it.

Airbnb's search team describes the same choice in a 2018 paper. Their ranking model orders listings by the guest's likelihood of booking. They compare models offline with NDCG, a score for how high the booked listing sat, then test them online in an A/B test on bookings.

The catch: bookings are rare

Bookings are a far rarer signal than clicks. The app sees 280 bookings a day but about 90,000 clicks. A rented flat also leaves the market for months, so one listing may get only one or two bookings a year.

Airbnb hit a milder form of this. Their 2018 paper notes that even the most popular listing can be booked at most 365 times a year. Giving the model a separate learned value for each listing ID mostly led to overfitting.

Overfitting means the model memorises its training data and does badly on new searches. So the model must learn from general features, such as price compared with the area, not from the history of each single flat.

The team also logs weaker signals that happen more often:

• Booking: the renter paid the token amount. The strongest signal, and the main training target.
• Visit request: the renter asked to see the flat. Common, and fairly close to intent.
• Owner contact: the renter called or messaged the owner. Useful, but noisier.
• Long view: the renter stayed on the listing page for over a minute. Weak, and easy for flashy photos to win.

Keep the weak signals as helpers, not the target. The same 2018 paper describes a model that predicted long views alongside bookings. Online, long views rose by a large margin, but bookings stayed neutral.

A search ranking pipeline works in stages. A fast search fetches the listings that fit the filters, up to a set limit. A model scores them, and a few business rules adjust the final page. Each stage handles fewer listings and spends more time on each one.

Scoring all 22,000 listings with a model for every search would waste time. So the first stage is a plain search index, like the index at the back of a book that maps words to pages. It answers which listings are in Whitefield, under Rs 9,000 and free on 1 December.

Hard filters live in this first stage. A flat that is already rented, or a men-only PG for a woman's search, must never reach the page. A model should never be trusted to learn rules like that.

The Elasticsearch documentation describes the same split. Its learning to rank feature, added in Elasticsearch 8.12.0, uses the trained model as a second-stage re-ranker: a simpler first step finds candidates, and the model reorders them.

Here is the design the team draws. Start at the renter's phone and follow the arrows.

A feature store is a shared place that keeps precomputed features ready, so the same numbers are used in training and at search time. The team updates slow features, like a listing's review score, every night. Fast ones, like whether the flat is still free, update within minutes.

The useful features describe the listing, the match between listing and query, and the renter's own situation. Match features often matter most, because the same flat is perfect for one renter and useless for another.

The Elasticsearch documentation groups ranking features in a similar way. It names document features such as a product's price, query features, and query-document features that describe a document in the context of one query. The table uses the rental app's examples.

Relative features usually beat raw ones. Rent of Rs 18,000 means little on its own. Rent 10 percent below similar flats in HSR Layout tells the model something real.

You turn each logged search that ended in a booking into training examples that say: this booked listing should have ranked above these other listings. The model learns to score listings so that booked ones come out on top.

Learning to rank is the name for this family of methods. There are three common ways to frame it:

• Pointwise: predict a booking chance for each listing on its own, like a yes or no question.
• Pairwise: given two listings from the same search, predict which one the renter preferred.
• Listwise: score a whole page at once and reward a better order overall.

Airbnb's 2018 paper shows one well-documented path. A neural network is a model built from layers of weighted sums, each followed by a simple non-linear step, that learns its own feature combinations. Their first simple one was pointwise: booked listings got a target of 1.0, the rest 0.0. The next version changed two things.

First, it trained on pairs of one booked and one not-booked listing from the same search. Second, it weighted each pair by how much swapping the two would change NDCG, the page's quality score. That second idea is called LambdaRank.

So moving a booked listing from position 2 to 1 counts for more than moving it from 10 to 9. The team builds its pairs with a small function. Each logged result has an ID, a position and a booked flag.

from types import SimpleNamespace as Row

def make_pairs(results):
    booked = [r for r in results if r.booked]
    if not booked:
        return []                        # no booking: nothing to learn
    winner = booked[0]
    seen_until = winner.position + 2     # a guess: renters glance 2 past their pick
    pairs = []
    for r in results:
        if r is winner or r.position > seen_until:
            continue
        pairs.append((winner.id, r.id))  # the renter saw r and chose winner
    return pairs

page = [Row(id=f"L{p}", position=p, booked=(p == 3)) for p in range(1, 21)]
print(make_pairs(page))

What just happened? The renter booked listing L3, so the function printed [('L3', 'L1'), ('L3', 'L2'), ('L3', 'L4'), ('L3', 'L5')]. Listings 1 and 2 sat above the pick and lost, and so did 4 and 5 just below it.

Listings from position 6 down were skipped. A listing at position 15 was probably never seen, so losing tells you nothing about it. The plus 2 is a guess; check it against how far your renters really scroll. In real training, each pair holds both listings' features, not just IDs.

For the model itself, the team starts with gradient boosted trees: many small decision trees, each one fixing the errors of the ones before. Airbnb's 2018 paper says a boosted tree model gave large booking gains before those gains levelled off and they moved to neural networks.

Elasticsearch's learning to rank feature also runs boosted tree models. The training loop runs every night. Follow it from the logs around to production, and notice the arrow back to the logs.

The simplest fix is to tell the model where each listing was shown during training, then give every listing the same position when it ranks live searches. The model can then separate a listing's real appeal from the boost of sitting at the top.

Position bias means people click and book what sits near the top, partly just because it sits there. Joachims, Swaminathan and Schnabel argued in 2016 that training directly on such biased clicks gives worse rankings, and built a fix.

Go back to the fish stall. The fish on top of the ice sells fastest, but partly because buyers see it first. If she judged each fish only by today's sales, whatever she put on top would always look like the best fish.

The rental logs have the same trap. Newest first put fresh listings on top, so fresh listings got more bookings. In the simulation later in this guide, about half of all bookings landed in the top 3 of 20 places.

Two ways to correct it

The first way is propensity weighting. A propensity is the chance that a position gets looked at. You estimate it, then give more weight to bookings at rarely seen positions. Joachims and colleagues built their method on this idea.

The second way is simpler to run. Airbnb's 2020 paper adds position as a feature during training. When ranking real searches, they set that feature to 0 for every listing. The paper notes this needs no separate propensity model and no shuffled results.

Adding the position feature with no control cut their offline NDCG by about 1.3 percent. To reduce how much the model depends on position, they set it to 0 on a random share of training rows. This is a form of dropout, and they used a rate of 0.15.

The paper reports a 0.7 percent gain in bookings in an online test. The engineer section below tests the same trick on logs from a newest-first sort.

The team adds one more habit. On 2 percent of searches (made up), it randomly shuffles results ranked 3 to 8. This small dose of randomness, called exploration, shows how listings do at positions they would not normally get.

You measure offline with NDCG, which scores how high the booked listing sat in each search, and online with an A/B test that counts real bookings. Offline tests are cheap and quick, but only the online test proves the change works.

NDCG stands for normalized discounted cumulative gain. Each relevant result earns points. Results lower down earn fewer points, using a log discount. The total is divided by the best possible score, so the result runs from 0 to 1.

Airbnb's 2018 paper used a simple version: the booked listing has a relevance of 1, and every other listing has 0. With one booked listing, NDCG becomes one short formula. The team replays last week's searches with the new model's order and scores each one.

import math

def ndcg_at_k(ranked_ids, booked_id, k=20):
    for i, listing_id in enumerate(ranked_ids[:k]):
        if listing_id == booked_id:
            return 1 / math.log2(i + 2)  # i is 0 for first place, so it scores 1.0
    return 0.0                           # booked listing missed the top k

def average_ndcg(replays):
    return sum(ndcg_at_k(order, booked) for order, booked in replays) / len(replays)

# each replay: the new model's order for one logged search, and what was booked
replays = [(["A", "B", "C"], "A"), (["D", "E", "F"], "E"), (["G", "H", "I"], "Z")]
for order, booked in replays:
    print(booked, round(ndcg_at_k(order, booked), 3))
print("average", round(average_ndcg(replays), 3))

What just happened? The first search booked the top listing and scored 1.0. The second booked the listing in second place and scored 0.631. The third booked a listing the new order did not show at all, so it scored 0.0, and the average is 0.544.

To check the maths, run python3 -c "import math; print(1/math.log2(3))". It prints 0.6309297535714575, the score for second place. Tenth place scores about 0.289. On scikit-learn 1.9.1, its ndcg_score function gave the same numbers for places 1, 2, 10 and 20.

The log discount is a convention for how fast attention falls down a page. It is not measured from your renters, so treat NDCG as a filter, not a verdict.

Why the online test still decides

Offline NDCG only replays listings the old ranking chose to show. It cannot know whether a renter would have booked a flat they never saw.

Airbnb's 2018 paper reports one change, adding hand-designed noise to features, that gained about 1 percent in offline NDCG. Online, it showed no statistically significant gain, meaning the difference was too small to tell apart from chance.

In an A/B test, the app sends a random 10 percent of renters to the new model. Each renter stays in one group for the whole test, so one booking is never split between two rankings. The main metric is bookings per search. The team also watches a few guardrail metrics, which are numbers that must not get worse:

• Search latency at the 99th percentile, the slowest 1 in 100 searches.
• Booking cancellations within 7 days, which catch listings that look good but disappoint.
• Share of bookings going to new listings, so they are not starved.
• Owner complaints about listings vanishing from results.

You rank new listings by estimating their missing history from similar listings nearby, and you keep results varied with light rules after the model. Both fix problems that a model scoring one listing at a time cannot see.

The cold start problem

A brand-new listing has no bookings, no visit requests and no reviews. That is the cold start problem. If those history features are simply set to 0, the model treats every new listing as a bad one, and it never gets shown long enough to prove otherwise.

Airbnb's 2020 paper estimated a new listing's engagement numbers by averaging listings nearby with the same guest capacity. The paper reports a 14 percent rise in bookings of new listings in an online test, and 0.38 percent more bookings overall.

The team copies the idea with rental details. A new single-sharing PG in Marathahalli borrows the average numbers of single-sharing PGs within 1 km, in the same rent band. In the diagram, notice that real history replaces the estimate once enough exists.

Keeping the page varied

A model that scores each listing on its own may fill the page with near copies. Ten rooms from one big PG building can all score high, and the renter sees no real choice.

Airbnb's diversity paper makes this point: ranking each listing alone ignores the result set as a whole, and diverse results give the user more choice. The team starts with simple rules and keeps them few:

• At most 2 listings from the same building in the top 10.
• Merge re-posted duplicates, matched by address and photos, and keep the oldest copy with its reviews.
• If the top 10 all sit in one rent band, pull up the best listing from another band.
• Drop any listing whose owner has not confirmed availability in 21 days.

The duplicate rule ends the morning re-posting trick, and the availability rule backs up the hard filter.

It learns the old ranking's habits. In a simulation we ran, a boosted tree ranker trained on newest-first logs leaned mostly on listing age, a feature we built to have no effect on bookings.

The world is made up and small. There are 5,000 listings, each with four quality features and an age in days. A listing's true appeal, its chance of being booked once seen, depends only on the four quality features.

The old ranking sorts each 20-listing page newest first. A renter looks at position k with a made-up chance of 1 divided by k, and books a seen listing with its appeal. About half of all bookings land in the top 3 places.

We simulate 30,000 searches, keep the roughly 15,000 that ended in a booking, and train two rankers with LightGBM, an open-source boosted tree library. Its default ranking objective, lambdarank, applies the LambdaRank idea from earlier to boosted trees. One model sees only the features.

The other model also sees the position each listing was shown at. At ranking time, every listing gets the same position. Both are then judged on 3,000 fresh pages against the true appeal, which a real team never gets to see.

Pause and predict: the naive model never sees position. Which input will it use to explain why the top listings got booked?

Run the simulation yourself

Install the versions we tested on 19 October 2026: pip install numpy==2.5.4 scikit-learn==1.9.1 lightgbm==4.7.0. The script ran in about 35 to 40 seconds on a quiet 2-core machine with Python 3.13, and longer on a busy one.

import numpy as np, lightgbm as lgb
from sklearn.metrics import ndcg_score

rng = np.random.default_rng(0)
n, page = 5000, 20
quality = rng.normal(size=(n, 4))          # price gap, reviews, photos, reply time
age = rng.uniform(0, 365, n)               # days since listed: no real effect
appeal = 1 / (1 + np.exp(2 - quality @ [-1.0, 0.8, 0.3, -0.5]))
X_all = np.column_stack([quality, age])

X, y, pos, groups = [], [], [], []
for _ in range(30000):                     # logs from the newest-first ranking
    page_ids = rng.choice(n, page, replace=False)
    page_ids = page_ids[np.argsort(age[page_ids])]
    seen = rng.random(page) < 1 / np.arange(1, page + 1)
    booked = seen & (rng.random(page) < appeal[page_ids])
    if booked.any():
        X.append(X_all[page_ids]); y.append(booked)
        pos.append(np.arange(1, page + 1)); groups.append(page)
X, y, pos = np.vstack(X), np.concatenate(y), np.concatenate(pos)

opts = dict(n_estimators=200, learning_rate=0.05, num_leaves=15,
            min_child_samples=50, verbose=-1, random_state=0)
naive = lgb.LGBMRanker(**opts).fit(X, y, group=groups)
fixed = lgb.LGBMRanker(**opts).fit(np.column_stack([X, pos]), y, group=groups)

def test(score):                           # fresh searches, judged on true appeal
    out = []
    for _ in range(3000):
        ids = rng.choice(n, page, replace=False)
        out.append(ndcg_score([appeal[ids]], [score(X_all[ids])], k=10))
    return round(float(np.mean(out)), 3)

print("newest first:", test(lambda x: -x[:, 4]))
print("naive model: ", test(naive.predict))
print("with position:", test(lambda x: fixed.predict(np.column_stack([x, np.ones(len(x))]))))
gain = naive.booster_.feature_importance("gain")
print("naive model's gain share on age:", round(float(gain[4] / gain.sum()), 2))

It printed 0.491 for newest first, 0.832 for the naive model and 0.985 with position. The last line showed 0.57 of the naive model's split gain on age. Split gain is how much each input improved the trees' decisions, so it shows what a tree model relies on.

What the numbers show

We repeated the run with five random seeds, the numbers that fix the made-up world, using a fuller script that also tracks how closely each score follows newness. Every row is a range across those five seeds.

What just happened? The naive model never saw position, so it found the next best explanation for why top listings got booked: they were new. Its scores followed newness closely, even though newness had no effect.

Given the position, the model let position explain the top-of-page boost, and age fell to 1 or 2 percent of split gain. Setting position to 0, 1, 5 or 20 at ranking time changed NDCG by about 0.01 at most.

Read these numbers as a demonstration, not a benchmark. The made-up world follows the exact assumption the position feature relies on: booking chance is attention times appeal. Real attention also depends on photos, device and query, so do not expect a gap this clean.

Not in our simulation. Hiding the position on 15 percent of training rows, the rate Airbnb used, brought the age habit straight back. It did so for the tree model and for a small neural network.

NDCG fell from about 0.985 to about 0.84, close to the naive model. Age's share of split gain jumped from 1 or 2 percent to 46 to 50 percent. Even a 5 percent rate pushed it to about a third in three seeds we tried.

The gain numbers show why. On rows where position is hidden, the trees need another way to explain why top listings were booked. Age is a perfect stand-in, because the old ranking sorted by age. Ranking with position set to 0 then switches on exactly that part of the model.

Ranking with position set to 1 instead of 0 helped the trees only partly, giving 0.869 to 0.886. So once dropout is on, the constant used at ranking matters, unlike in the run without dropout.

To check this was not a tree quirk, we trained a small neural network, scikit-learn's MLPClassifier, on the same logs, scoring each listing on its own. With position as a feature, set to 1 at ranking, it reached 0.949 to 0.977 across three seeds. With 15 percent dropout and position 0 at ranking, it fell to 0.822 to 0.834, back near its naive level of 0.831 to 0.849.

This does not show the Airbnb paper was wrong. Our best guess at the difference is the logs: Airbnb's results were already ranked by learned models, not sorted on one feature.

It shows that a bias fix tuned on one kind of log can backfire on another. Re-test it on your own logs, ideally with an exploration slice where you know the order was random.

Explain it as a pipeline with a clear goal: fetch candidates fast, score them by the outcome the business cares about, and prove each change with real users. Walk through it in this order.

1. 1

   Pin down the goal

   Ask what success means. For a rental app, say bookings per search, with cancellations and latency as guardrails.

2. 2

   Size the problem

   Give rough numbers: listings, searches a day, candidates per search and a latency budget. This sets how many candidates you can score.

3. 3

   Draw the stages

   Search index with hard filters, feature store, ranking model, business rules, and an event queue logging every impression.

4. 4

   Choose labels and features

   Bookings as the main label, visit requests as helpers. Name listing, match, renter and context features, and warn about future leaks.

5. 5

   Train and correct bias

   Pairwise training from logged searches, a simple boosted tree model first, and position as a feature set to one value at ranking time.

6. 6

   Measure in two steps

   NDCG offline to filter ideas, then an A/B test on bookings to decide.

7. 7

   Handle the edges

   Cold start for new listings, duplicates, variety on the page, stale availability and a small exploration slice.