Design a Semantic Search Engine: Find Jobs by Meaning

They get zero or wrong results, because seekers describe work in their own words while employers write posts in different ones. Keyword search only matches words, so the right job can exist and still stay hidden. This guide covers semantic search system design: building search that matches meaning instead.

Picture a common setup. A job search app for retail, logistics and office work in Indore has about 1.8 lakh (180,000) live job posts and about 60,000 searches a day, mostly on low-cost phones. The setup is made up, but the problem is real.

The team pulls a week of search logs and finds that 22 in every 100 searches return zero results (made up). Reading a few hundred of them shows a pattern.

• Someone types maal loading ka kaam. The matching posts say warehouse associate and material handler, so nothing comes back.
• Someone types computer typing job part time. The posts say data entry operator, 4 hours. Again, zero results.
• Someone types delivery boy with bike. The posts say last mile rider, own two-wheeler. Keyword search finds only the posts that copied the phrase delivery boy.
• Someone types shop helper Vijay Nagar. Keyword search matches the word helper in a hospital canteen post across the city.

Each of these people has a job waiting in the database. The search cannot see it, because it compares spellings, not needs. The rest of this guide designs the search that can, and shows where each part breaks.

You need five words: embedding, vector, cosine similarity, index and recall. Each is explained here with the job app's own example.

Embedding. An embedding is a fixed-length list of numbers that a trained model, the embedding model, produces from a piece of text. A good model gives texts with similar meaning similar lists.

Vector and dimension. A vector is that list treated as a point in space, and each number in it is one dimension. The job app's model outputs 384 numbers per text. You cannot read what each number means, and you do not need to.

Cosine similarity. This score says how closely two vectors point the same way. It runs from minus 1 to 1, and higher means closer in meaning.

Index. An index is a data structure built ahead of time so that lookups are fast. A vector index finds the stored vectors closest to a query vector.

Recall. Recall at 10 is the share of the true best 10 matches that a search actually returns. A recall of 0.9 means 9 of the real top 10 came back.

Semantic search system design is planning a search system that matches what a query means, not just its words. It covers turning text into vectors, storing them, finding the closest ones fast, and checking the results.

Think about a labour naka, the roadside spot where workers wait at dawn for a day's work. A builder walks up and says he needs someone for the walls. The contractor does not look for a worker with walls written on his shirt.

He knows the builder means a bricklayer or a plasterer, not a painter. He matches the need to the skill even though the words differ. The builder's request is the query, the workers are the job posts, and the contractor's sense of who fits is the embedding model.

Where the picture breaks: the contractor can ask a follow-up question and knows exact licences by heart. An embedding model does neither. It returns its closest guesses, even for nonsense, and it can blur exact names.

A full design has two halves. The indexing path runs in the background and prepares every job post. The query path runs on each search and must answer in a fraction of a second. You also need a way to measure quality.

Write the targets down before drawing anything. For the job app: answer 95 in every 100 searches within 300 milliseconds, and make a new post searchable within 2 minutes (both made up). A filled job must vanish just as fast, so nobody applies for a job that is gone.

Once text is a vector, comparing meaning becomes simple maths: score how closely the query vector points towards each job vector, then keep the highest scores.

If both vectors are first scaled to length 1, a plain dot product (multiply matching numbers, then add them up) equals cosine similarity. Here is the simplest possible semantic search, with no index at all. It runs as shown with numpy.

import numpy as np

def top_jobs(query_vec, job_vecs, job_ids, k=10):
    # Scale every vector to length 1, so a dot product is cosine similarity
    q = query_vec / np.linalg.norm(query_vec)
    jobs = job_vecs / np.linalg.norm(job_vecs, axis=1, keepdims=True)
    scores = jobs @ q                      # one score per job post
    best = np.argsort(-scores)[:k]         # highest scores first
    return [(job_ids[i], round(float(scores[i]), 3)) for i in best]

job_vecs = np.array([[1.0, 0.0], [0.6, 0.8], [0.0, 1.0], [-1.0, 0.0]])
job_ids = ["warehouse", "loading", "driver", "tailor"]
print(top_jobs(np.array([3.0, 1.0]), job_vecs, job_ids, k=3))

What just happened? The query points mostly along the first direction, like the warehouse post. Running the file prints [('warehouse', 0.949), ('loading', 0.822), ('driver', 0.316)]. The tailor post points the opposite way, scores minus 0.949, and is left out of the top 3.

This is exact, or brute force, search because it scores every vector. For the job app, one search costs 1.8 lakh times 384, about 6.9 crore (69 million) multiplications. The vectors take 276 MB of memory as 4-byte floats.

You can check the memory maths yourself. Run python3 -c "print(180_000 * 384 * 4)" and it prints 276480000, which is bytes, so about 276 MB.

Is exact search too slow? We timed it on a shared 2-core test machine with 1.8 lakh made-up vectors. With the vectors scaled once in advance and only the top 10 picked out, one query took 41 to 150 ms. Scaling them inside every call, as the sample does for clarity, took 0.6 to 1.2 seconds.

For the job app, that is fast enough at the average load. 60,000 searches a day is under 1 a second, and that machine answers 7 to 24 a second, one at a time.

The pressure comes from elsewhere. A 150 ms query uses half the 300 ms target before reranking starts. Busy hours run above the average, and exact search time grows in step with the post count.

Why embed posts separately at all? A 2019 paper on sentence embeddings by Reimers and Gurevych measured the alternative. A model that reads two sentences together needed about 65 hours to find the most similar pair among 10,000 sentences. Separate embeddings did it in about 5 seconds.

So each job post is embedded once and stored. At search time only the short query needs the model. This setup, where queries and documents are encoded alone and never read together, is called a bi-encoder or two-tower model.

The indexing pipeline watches the job database for changes, embeds each new or edited post, and writes the vector into the search indexes. It also removes posts that are filled or expired.

The app does not write to the search index directly. Every change to a job post goes out as an event on a message queue, a list of messages that workers read in order. That way a slow embedding step never slows down an employer saving a post.

In the diagram, start at the employer and follow both branches from the queue. One feeds the vector index, the other the keyword index you will meet later.

What text should a job post become? Build one short string from the title, the main duties, the area and the shift. Leave out the salary and the phone number. Numbers like these add little meaning and work better as exact filters.

• Store filter fields next to each vector: area, pay range, shift, job type and posting date.
• Use the job id as the vector's id. In hnswlib 0.8.0, adding a vector with an existing id updates it in place; we checked that the count stays the same.
• Send a delete event the moment a job is filled, and remove its vector from the index.
• Keep the model name and version on every vector, so you know which ones need rebuilding after an upgrade.

Approximate nearest neighbour (ANN) search finds vectors very close to the query without checking every vector. It is usually right, not always right, and that small gap buys a large speed-up.

A widely used ANN index is HNSW, short for hierarchical navigable small world graphs, from a 2016 paper. Each vector becomes a point in a graph, linked to a handful of close neighbours. The graph has layers, like maps at different zoom levels.

The top layer holds only a few points with long links. A search starts there and jumps towards the right area. Then it drops a layer and takes shorter steps. The bottom layer holds every point, and there it collects the closest ones it can find.

The paper reports that this layered design allows logarithmic complexity scaling. Roughly, each tenfold growth in the collection adds a similar number of extra steps, not ten times the work. That is the paper's finding on its test data, and the price is recall.

• Links per point, called M. More links give better recall but use more memory and take longer to build.
• Search depth, called ef. It is how many candidates the search keeps in hand at the bottom layer. Higher means better recall and slower searches.
• Memory. HNSW is fastest when the whole graph sits in RAM. Each vector costs its raw 1,536 bytes plus its links.

Tune these by measuring, not guessing. Run 1,000 saved queries through exact search to get the true top 10. Run the same queries through HNSW and compute recall. Pick the fastest setting that keeps recall where you need it.

Filters change the plan

Most job searches carry filters, like night shift jobs within 5 km of Palasia. A tempting shortcut is to fetch the 20 nearest jobs and then drop those that fail the filter. That is called post-filtering, and it is the wrong first try.

Pause and predict: if 1 job in 10 passes the filter, how many of the 20 nearest survive? In our test on 50,000 made-up vectors, the median was 2 and the worst was 0.

The fix is to choose the method by how many jobs pass. When few pass, exact search over just those is cheap. When many pass, use the index and apply the filter during the search. Follow the question box and its two answers.

Each fails where the other is strong. Semantic search handles different words for the same need. Keyword search handles exact names, codes and rare terms that embeddings blur.

Back at the naka, a request for someone for the walls needs the contractor's sense of meaning. A request for a forklift operator with a licence does not. He finds exactly that, not someone close enough.

The job app sees the same split. A search for Tally ERP, an accounting software name, may land near loosely related office jobs with vectors alone. A keyword index, which maps each word to the jobs that contain it, matches the exact name.

Research shows both sides. BEIR, a 2021 benchmark across 18 datasets, found BM25 a strong baseline across those datasets. BM25 is a classic formula that scores a document by how often the query words appear in it and how rare those words are overall.

The same benchmark found dense retrievers, meaning embedding-based search models, efficient but often behind the other approaches on datasets they were not trained for. A 2020 dense passage retrieval paper showed the other side.

Its trained bi-encoder beat a strong BM25 system by 9 to 19 points of top-20 accuracy across open-domain question sets. Which wins depends on your data, so you measure on your own queries.

Hybrid search runs both and merges the two ranked lists. The raw scores do not compare, since a BM25 score and a cosine score use different scales. So merge by rank instead, with reciprocal rank fusion (RRF), from a 2009 paper.

def fuse(keyword_ids, vector_ids, k=60):
    scores = {}
    for ranked in (keyword_ids, vector_ids):
        for rank, job_id in enumerate(ranked, start=1):
            scores[job_id] = scores.get(job_id, 0.0) + 1 / (k + rank)
    return sorted(scores, key=scores.get, reverse=True)

keyword = ["tally-accountant", "billing-clerk", "data-entry"]
vector = ["data-entry", "tally-accountant", "office-assistant"]
print(fuse(keyword, vector))

Each list gives a job 1 divided by (k plus its rank), and the shares add up. The file prints ['tally-accountant', 'data-entry', 'billing-clerk', 'office-assistant']. The two jobs found by both lists rise above the jobs found by one.

The constant k is 60 in the original paper, and it is Elasticsearch's default as of 19 October 2026. With k at 60, rank 1 earns 0.0164 and rank 10 earns 0.0143. With k at 0, rank 1 would earn ten times rank 10 and swamp everything.

Reranking takes the top few dozen results from the fast first stage and scores them again with a slower, more careful model. Only that short list pays the higher cost, so the search stays fast.

The careful model is a cross-encoder: it reads the query and one job post together and outputs one relevance score. Reading both at once lets it weigh details such as part time in the query against full time in the post. A bi-encoder compresses each side alone first.

Remember the 65 hours from that 2019 sentence-embedding paper. That cost is why a cross-encoder never runs on all 1.8 lakh posts. The job app runs it on the top 50 fused results, which takes about 60 milliseconds on its hardware (made up).

The reranker is also where business rules join in. The app adds small boosts for verified employers and for posts from the last week, and pushes down posts with no salary listed. Learning the weights for signals like these from click data, instead of setting them by hand, is called learning to rank.

Now put it together. This diagram shows one search from tap to screen. Each box is one step, and its note says which part talks to which.

Split the 300 millisecond budget across these steps. A made-up plan: 20 for embedding the query, 60 for both searches, 5 for fusion, 60 for reranking, and the rest for the network. Short queries like delivery job repeat a lot, so caching their results saves real work.

You know it works when two kinds of numbers improve: offline scores on labelled queries, and real user behaviour after launch. Neither alone is enough.

Offline first. The team takes 500 real queries and marks, for each one, which job posts are good matches. That is a labelled set. Every design change runs against it before it ships.

• Recall at 20: of the good jobs for a query, the share that appears in the top 20. Unlike the index recall defined earlier, it is measured against human labels, not exact search. An index can match exact search perfectly and still score low here. It tells you whether the first stage finds the right jobs at all.
• nDCG, short for normalised discounted cumulative gain: a score from 0 to 1 that rewards putting the best jobs near the top. It tells you whether the reranker orders them well.
• Zero-result rate: the share of searches that return nothing. It was the problem that started this project.

Recall needs only a few lines. This function and its example run as shown.

def recall_at_k(ranked_ids, good_ids, k):
    # Share of the good jobs that appear in the top k results
    return len(set(ranked_ids[:k]) & set(good_ids)) / len(good_ids)

ranked = ["j7", "j2", "j9", "j4"]
good = {"j2", "j4", "j5"}
print(round(recall_at_k(ranked, good, 2), 3), round(recall_at_k(ranked, good, 4), 3))

It prints 0.333 0.667. Three jobs are good; the top 2 hold one of them, and the top 4 hold two.

Then online. Run an A/B test, where half the users get the new search and half the old, for two weeks. Watch what matters to the business: how many searches lead to a tap on a job, and how many lead to an application.

The labelled set must grow with real traffic. Add fresh queries every month, especially Hinglish ones and ones that got no clicks. Search that looked good on old queries can quietly get worse as job types and words change.

Semantic search fails more quietly than keyword search. It almost always returns something, so a wrong result looks like a right one unless you check.

• Confident nonsense. A gibberish query still has nearest neighbours. Guard: a minimum similarity score, tuned on the labelled set, below which you show a clear no good match message.
• Lost exact terms. A licence type or software name gets blurred. Guard: hybrid search with a keyword index.
• Mixed model versions. After an upgrade, half the vectors come from the old model. Guard: a version tag on every vector, and a full re-embed into a new index before switching.
• Stale jobs. A filled job keeps showing. Guard: delete events, plus a final open-job check before showing results.
• Language gaps. A model chosen on English text may handle Hindi typed in Latin letters badly. Guard: put plenty of such queries in the labelled set before choosing a model.

Model upgrades deserve a plan of their own. Build the new index beside the old one, fill it with new vectors, test it, then switch the query path in one step. Keep the old index until you are sure you will not roll back.

Higher M and higher ef both raise recall and cost time; M also costs memory. The surprise is the default: hnswlib, a widely used HNSW library, searches with ef set to 10 unless you change it.

We measured this on hnswlib 0.8.0, the latest release on PyPI as of 19 October 2026. The data was 50,000 made-up 384-number vectors in 1,000 clusters, with 500 fresh queries. These are not real job embeddings, so treat the numbers as a shape.

What just happened? With M at 8, the default depth missed almost half of the true top 10. Raising ef to 64 fixed recall for a few tenths of a millisecond. Exact search took 8 to 16 ms here, more than ten times longer than any HNSW setting in the table.

The library clamps the depth to at least k, the number of results you ask for. We read this in its source, in hnswalg.h, where the search uses the larger of ef and k. So ef 10 with k 10 means no spare candidates at all.

Memory follows a formula from the same file. Each vector stores its raw numbers, 2 times M links of 4 bytes each, a 4-byte link count and an 8-byte id. For 384 numbers and M 16, that is 1,676 bytes.

A few vectors also sit on upper layers, which brought the pickled index to 1,696 bytes per vector. The running process grew by 1,849 bytes per vector, since it also keeps an id lookup table, a lock and a level for each vector (hnswalg.h lines 43 to 60).

One more trap: save_index and load_index do not keep ef. We set ef to 64, saved, loaded, and read back 10; the GitHub README says the same. A pickled copy kept 64.

To check your own index, run it on a sample, pickle it with Python's pickle module, and divide the byte length by the vector count. Expect a little more than 4 bytes times the dimensions.

In hnswlib, both a strict filter and a pile of deleted vectors make each search walk more of the graph. Every result passes the filter and no deleted vector comes back. But the time per query grows, recall can slip, and a very strict filter can cost more than exact search.

Strict filters make the walk longer

hnswlib's knn_query accepts a filter, a function that returns True for ids allowed in the results. The search still steps through failing vectors, but only counts passing ones. It can stop only once it holds ef passing candidates, or when it runs out of graph to walk.

So the fewer jobs pass, the further it walks. This sample counts how many vectors the filter checks for one query. It runs in a few seconds with hnswlib 0.8.0 and numpy.

import numpy as np, hnswlib  # pip install hnswlib==0.8.0 numpy

rng = np.random.default_rng(0)
vecs = rng.standard_normal((20_000, 384)).astype(np.float32)
index = hnswlib.Index(space="cosine", dim=384)
index.init_index(max_elements=20_000, M=16, ef_construction=200)
index.add_items(vecs, np.arange(20_000))
index.set_ef(64)

for share in (1.0, 0.01):
    allowed = rng.random(20_000) < share   # e.g. night shift jobs only
    checked = 0
    def passes(job_id):
        global checked
        checked += 1
        return bool(allowed[job_id])
    ids, _ = index.knn_query(vecs[:1], k=10, filter=passes, num_threads=1)
    print(f"{share:.0%} pass: checked {checked} of 20000, all pass: {allowed[ids].all()}")

Two runs printed about 230 checks when every job passes and about 11,600 of 20,000 when 1 in 100 passes. Your counts will differ a little, because add_items builds the graph with several threads by default.

At 1 in 1,000, only about 50 jobs passed, fewer than the 64 that ef asks for. The search could never fill its list, so it walked the whole graph, slower than plain brute force. This is why the decision diagram counts matching jobs first. The hnswlib README does not describe this cost as of 19 October 2026, so treat it as undocumented and pin your version.

Part of that time is Python calling the filter once per check, about 0.6 to 1 microsecond each in our runs. At 10%, that is 1 to 2 ms of the 5 to 6 ms. Without it, the index would still need about 3.5 to 5 ms, so the longer walk is why exact search won there. The hnswlib README itself warns that filtered search is slow in Python with several threads and recommends one thread.

If fewer than k ids pass in total, knn_query raises a RuntimeError saying it cannot return the results in a contiguous 2D array. Count first, and never ask for more results than pass.

Deleted vectors stay in the graph

hnswlib's mark_deleted hides a vector from results but keeps it in the graph, so searches still step through it. In one run, we deleted growing shares of the same index and timed the same queries.

• 0% deleted: 0.39 ms per query, recall at 10 of 1.000.
• 50% deleted: 0.80 ms per query, recall 1.000.
• 90% deleted: 3.77 ms per query, recall 0.924. No deleted vector was ever returned.
• The element count stayed at 50,000 throughout, so memory is not freed either.

Two ways out exist. Create the index with allow_replace_deleted set to True, then add new vectors with replace_deleted set to True; they reuse deleted slots, as our test confirmed. Or rebuild the index from the database on a schedule.

Other libraries differ. In faiss 1.15.1, another popular vector search library, the HNSW index cannot remove vectors at all; remove_ids raised an error in our test. Check your own library before you design deletes.

Explain it as two paths, indexing and querying, then add filters, hybrid search, reranking and measurement, with a reason for each part.

1. 1

   Clarify the goal

   Ask what is searched, how many items, how many searches a second, the time budget, and how fast new and removed items must show.

2. 2

   Draw the indexing path

   Database, change events, embedding workers, and a vector index with filter fields beside each vector. Mention deletes and model versions.

3. 3

   Draw the query path

   Embed the query with the same model, run an ANN search, and pick exact or filtered index search by how many items pass the filters.

4. 4

   Add keyword search

   Run a keyword index alongside, merge with reciprocal rank fusion, and give one example where each search fails alone.

5. 5

   Add a reranker

   Rescore the top 50 with a cross-encoder and business rules. Explain why it cannot run on the whole collection.

6. 6

   Do the maths

   Raw vector memory is items times dimensions times 4 bytes; add HNSW links on top. Split the time budget across embedding, search, fusion and reranking.

7. 7

   Say how you measure

   Recall and nDCG on a labelled set offline, then click and apply rates in an A/B test.