Embeddings Explained: Meaning as Numbers

Keyword search only finds items that share the searcher's exact words, so it misses items described in other words. Embeddings close that gap by comparing meaning instead of spelling.

Picture a common setup. An online marketplace sells second-hand books from about 2,000 sellers, with around 40,000 listings in English and Marathi. The setup is made up, but the problem is real.

Its search box matches words. A month of search logs shows that 18 percent of searches return nothing at all. Reading a few hundred of them, the team finds four patterns.

• A buyer types detective stories in old Bombay. The right book is listed as crime fiction set in 1960s Mumbai. Not one word matches, so the buyer sees nothing.
• One seller lists the same novel twice: once as a used paperback, and once as a second-hand copy with a torn cover. Buyers see both and get confused.
• A buyer searches in Marathi for detective stories. The English crime novels never show up.
• Buyers who loved a book ask for something similar, and today a person answers those requests by hand.

The team's first fix is the naive one: a synonym list. Bombay means Mumbai, and detective means crime. After 300 rules most failed searches still fail, because people describe the same thing in endless ways.

What the team needs is a way to compare meaning, not spelling. That is what an embedding gives you. This guide follows the marketplace as it adds embeddings, and shows where they fail.

An embedding is a list of numbers that stands for the meaning of a piece of text. Texts with similar meanings get similar lists, even when they share no words at all.

Think about ragas. Two songs in raga Yaman can have different lyrics and still feel related to anyone who listens. Two songs that both use the word chand (moon) can feel nothing alike.

Keyword search hears only the lyrics. An embedding tries to hear the raga. Where the picture breaks: a raga has rules a musician can name, while an embedding's sense of similarity is learned from examples, and nobody can read its rules back out.

So detective stories in old Bombay and crime fiction set in 1960s Mumbai get lists of numbers that sit close together. A romance novel set in Mumbai gets a list that sits further away, even though it shares the city.

You can embed a word, a sentence, a book listing or a whole page. Images and audio can be embedded too, but this guide sticks to text.

You may have read that a large language model turns each token (a small piece of text) into an embedding inside itself. That is the same idea, used as the model's first step, and our guide on how LLMs work covers it.

Here we mean text embeddings: one list of numbers for a whole piece of text, from a model you call directly.

Four words you will see often

Vector. A vector is an ordered list of numbers, such as 0.12, -0.40, 0.07. In this guide, vector and embedding mean the same thing.

Dimension. A dimension is one position in that list. A vector of 384 numbers has 384 dimensions.

Embedding model. An embedding model is a model trained to turn text into a vector of a fixed length. It does not write answers or chat. It only returns numbers.

Similarity score. A similarity score says how close two vectors are. The marketplace uses cosine similarity, which you will work out by hand further down.

Here is the whole idea at the marketplace in one picture.

A vector is a point in space, and each dimension is one direction that point can move along. Text embeddings commonly have a few hundred to a few thousand dimensions.

For scale: in a 2022 paper that benchmarked 33 text embedding models, one stored embedding took between 1.2 kB and 16.4 kB. At 4 bytes per number, that is about 300 to 4,100 numbers.

Picture describing every book with just three numbers between 0 and 1. They say how much it is about crime, how much it is set in Mumbai, and how much it is a romance. These three sliders are made up to show the idea.

A Mumbai crime novel might be 0.8, 0.9 and 0.0. A search for detective stories in old Bombay might be 0.9, 0.8 and 0.1, which lands right next to it. A Mumbai love story lands further away.

Every vector from one model has the same length, whatever goes in. A three-word search and a 200-word listing both become 384 numbers. That fixed length is what lets you compare any two of them.

More dimensions can hold finer shades of meaning. The price is more memory per vector and slower comparisons. For short texts like book listings, a smaller model is often good enough, and you find out by testing.

Every model also has a maximum input length, counted in tokens. Longer text may be cut off without any error, as an engineer section below shows. Long documents are usually split into pieces before embedding, while a book listing fits easily.

You usually measure it with cosine similarity, which checks whether two vectors point in the same direction. The score runs from -1 to 1, and higher means more alike.

Cosine similarity ignores how long the vectors are and looks only at their direction. When both vectors are scaled to a length of exactly 1, cosine similarity equals the dot product (multiply matching numbers and add them up).

Here is a tiny worked example with the three book sliders, so you can follow the maths. Pause and predict first: the romance listing shares the Mumbai number with the search. Will it score near 0.99 or near 0?

import math

def cosine(a, b):
    dot = sum(x * y for x, y in zip(a, b))
    size_a = math.sqrt(sum(x * x for x in a))
    size_b = math.sqrt(sum(y * y for y in b))
    return dot / (size_a * size_b)

# three made-up sliders: crime, set in Mumbai, romance
query = [0.9, 0.8, 0.1]     # detective stories in old Bombay
crime = [0.8, 0.9, 0.0]     # crime fiction set in 1960s Mumbai
romance = [0.1, 0.9, 0.9]   # romantic novel set in Mumbai

print(round(cosine(query, crime), 2))     # 0.99
print(round(cosine(query, romance), 2))   # 0.58

What just happened? The dot line multiplies matching numbers and adds them up. The two size lines measure each vector's length, and dividing by both leaves only the direction.

The crime listing scores 0.99, and the romance listing lands in between at 0.58. Sharing one strong slider pulled it up, but the crime and romance sliders pulled it down.

Check the crime score by hand. The dot product is 0.72 plus 0.72 plus 0, which is 1.44. The two lengths are about 1.21 and 1.20, which multiply to about 1.455. And 1.44 divided by 1.455 is about 0.99.

Scores only mean something next to each other. There is no universal cut-off that means similar, because the same unrelated pair can score near 0 on one model and well above 0 on another.

One reason is anisotropy: pre-trained language models, first trained on large amounts of general text, tend to squeeze their vectors into a narrow cone. A 2021 paper on simple contrastive learning of sentence embeddings reports that contrastive training spreads them out more evenly. Read real pairs from your own data before you pick a threshold (the cut-off score you act on).

Some systems use the dot product or straight-line distance instead of cosine similarity. Use the measure your model was trained with. Its documentation usually says which one.

They learn through contrastive training: the model sees pairs of texts that belong together and is nudged to score them higher than unrelated texts. Repeat that over millions of pairs, and meaning turns into distance.

Training data comes as matching pairs. A question and its answer. A title and its article. A sentence and its translation. For the marketplace, a search and the listing that buyer then bought.

The wrong matches come free. Training works in batches, groups of pairs scored together in one training step. In each batch, the texts from the other pairs act as wrong matches. With a batch of 64 pairs, each true pair is scored against 63 wrong ones, and the model is pushed to rank the true pair first.

Wrong matches that look close are called hard negatives, and they teach a lot. A 2020 paper on dense passage retrieval, which means finding passages by embedding, added one per question: a keyword-search result that did not contain the answer.

At a batch size of 128, that one change raised top-20 accuracy on its development questions from 73.0 to 78.0 percent. Development questions are held back from training and used only to check and tune the model. Top-20 accuracy means how often a correct passage appears in the first 20 results.

Here is one round of that training as a picture. Follow the arrow from the check back to the next batch. That loop runs a huge number of times before a model is ready.

The 2021 contrastive learning paper shows two ways to get pairs. Its unsupervised version pairs each sentence with itself, changed only by dropout, a little random noise inside the model.

Its supervised version uses sentence pairs where one follows from the other as matches, and contradicting sentences as hard negatives.

The 2020 passage retrieval paper also shows the payoff. Its embeddings, learned from a small set of questions and passages, beat BM25, a classic keyword-ranking method, by 9 to 19 points of top-20 accuracy.

Learning vectors from text is older than either paper. A 2013 paper trained vectors for single words and reported learning good ones from 1.6 billion words in under a day. Today's text embedding models embed whole sentences and passages instead.

The marketplace team does not train a model. It starts with a ready-made open model. Later it could fine-tune it, which means training it a little more on its own pairs of searches and bought listings.

With one vector per item, you can build search by meaning, duplicate checks, grouping and recommendations. All four come down to the same move: find the vectors closest to this one.

Search by meaning

Semantic search, which means search by meaning, embeds the buyer's query and returns the listings with the highest scores. The team embeds all 40,000 listings once and saves the vectors. Each search then needs one embedding call and one round of scoring.

At 40,000 listings, scoring every vector is quick, and an engineer section below times it. At millions, you need an index that finds close vectors without checking each one, which our guide on vector database architecture covers.

Finding duplicates

Two listings that say the same thing in different words get vectors that almost overlap. The team checks each new listing against that seller's earlier ones. The code below runs as it is, with three-slider vectors standing in for real ones.

import math

def cosine(a, b):
    dot = sum(x * y for x, y in zip(a, b))
    return dot / (math.sqrt(sum(x * x for x in a)) * math.sqrt(sum(y * y for y in b)))

DUPLICATE_SCORE = 0.95   # made up: set yours by reading real pairs

def find_near_duplicates(new_vector, seller_listings):
    matches = []
    for listing_id, old_vector in seller_listings:
        score = cosine(new_vector, old_vector)
        if score >= DUPLICATE_SCORE:
            matches.append((round(score, 3), listing_id))
    return sorted(matches, reverse=True)

seller_listings = [("L1", [0.8, 0.9, 0.0]), ("L2", [0.1, 0.9, 0.9])]
print(find_near_duplicates([0.82, 0.88, 0.05], seller_listings))
# [(0.999, 'L1')]

Two lines matter. The 0.95 threshold must come from reading real pairs from your own model, not from a rule of thumb. The loop covers only the same seller, because two sellers offering the same book is normal business.

The function only flags matches. The seller sees a prompt asking whether this is the same copy, and decides.

Grouping and recommendations

Clustering means grouping vectors that sit close together, without saying in advance what the groups are. Run over all listings, it can surface groups nobody had named, such as Pune local history. A person still names and checks each group.

For a More like this panel, the team averages the vectors of the last three books a buyer viewed. It then shows the closest listings the buyer has not seen yet. It works best for buyers who already have some history.

Yes, if you use a multilingual embedding model, which places a sentence and its translation close together. A Marathi search can then find an English listing with the same meaning.

A 2020 paper shows one common way to build such a model. It trains a new model on translated sentences to copy an existing one-language model, so a translation lands where the original sentence did. The authors show it working for more than 50 languages.

About a third of the marketplace's listings are in Marathi, and many buyers search in Marathi or Hindi. With a multilingual model, a Marathi search for detective stories can also bring up English crime fiction.

Test it on how your users really type, including Marathi or Hindi typed in English letters. Collect around 100 real searches in each style, and check the results by hand before you trust the scores.

Embeddings go wrong when meaning depends on small words like not, on exact numbers, or on words the model rarely saw in training. They capture what a text is about far better than its exact details.

• Negation. A search for history books that are not about wars can still pull up war histories. The text is about wars either way.
• Numbers. Under Rs 200 and under Rs 2,000 look almost the same as text, and nothing in a similarity score does maths on prices. Do not expect the vectors to keep them apart.
• Codes. ISBNs (the 10- or 13-digit number that identifies a book edition) and internal shelf codes carry little meaning a general model has learned. Two codes one digit apart may still score high.
• Business words. Collectors care about a first edition, and a general model may treat it like any old copy. Such words may need fine-tuning or exact matching.

Negation is not just a marketplace worry. A 2025 study of general-purpose text embedding models found they often rate a negated sentence as similar to the original.

The marketplace's search service does exactly that. A small parser spots phrases like under Rs 200 and turns them into a price filter. For searches with not, it drops listings whose category matches the excluded word.

Store each vector next to the item's id, along with the name and version of the model that made it. Vectors from two independently trained models cannot be compared, even at the same length, so that label matters.

Embed new and edited listings in a background worker, not while the seller waits. In the diagram below, the vector store returns the closest 50 listings, and the search service filters them down to 10. If a strict price filter leaves fewer than 10, the service asks for more candidates, or the vector store applies the filter before ranking.

Changing the model is a migration, not a config change. Re-embed every item with the new model, build the new set beside the old one, and switch searches over only when it is complete.

On vectors scaled to length 1, they are tied by one formula: squared straight-line distance equals 2 minus 2 times the cosine. So highest cosine, highest dot product and smallest distance give the same order.

We checked this on 5,000 random length-1 vectors of 384 numbers. The top 100 came out in the same order all three ways. When we gave the vectors lengths between 0.5 and 2, only 4 of the dot product's top 10 matched cosine's top 10.

So the dot product is safe as a stand-in for cosine only after scaling. Some models and tools scale for you, and some do not. Check one vector's length; it should print 1.0.

import numpy as np

v = np.array([0.9, 0.8, 0.1])
print(round(float(np.linalg.norm(v)), 3))      # 1.208: not length 1

unit = v / np.linalg.norm(v)
print(round(float(np.linalg.norm(unit)), 3))   # 1.0

Averaging, as in the More like this panel, also breaks the length. Three averaged length-1 vectors gave a length of 0.598 in our test. With the dot product, every score shrinks by that factor: the ranking holds, but a fixed threshold breaks. Cosine similarity is not affected.

Fast enough that you do not need an index yet. In our test, scoring 40,000 vectors of 384 numbers and picking the top 10 took a median of 1.4 ms. That ran on NumPy 2.3.4 on an idle 2-core 2.1 GHz server whose cache holds the whole 61 MB matrix.

Run it yourself. The listings here are random, and the query is listing 123 with a little noise added, so you know which result should win.

import time
import numpy as np

rng = np.random.default_rng(7)
listings = rng.normal(size=(40_000, 384)).astype(np.float32)
listings /= np.linalg.norm(listings, axis=1, keepdims=True)
query = listings[123] + 0.01 * rng.normal(size=384).astype(np.float32)
query /= np.linalg.norm(query)

times = []
for _ in range(21):
    start = time.perf_counter()
    scores = listings @ query
    top = np.argpartition(-scores, 10)[:10]
    top = top[np.argsort(-scores[top])]
    times.append(time.perf_counter() - start)
print(f"median {sorted(times)[10] * 1000:.1f} ms")
print(top[0], round(float(scores[top[0]]), 3))
print(f"{listings.nbytes / 1e6:.2f} MB")

Expect a median of a few milliseconds, then 123 0.983, then 61.44 MB. Timing depends on the machine: the same server took 20 to 36 ms while busy with other jobs. That last line is the storage maths: 40,000 times 384 times 4 bytes is 61,440,000 bytes.

Watch the units. 61.44 MB is decimal megabytes, and the same size shows as 58.59 MiB in tools that count in powers of 2. Storing 16-bit numbers instead halves it to 30.72 MB. Scored in 32-bit, these random vectors' scores moved by under 0.0001.

A full scan grows in step with the item count, so 100 times the listings means about 100 times the work. That is when an index starts to earn its place.

Two library defaults change results without any error: cutting long text, and not scaling vectors to length 1. We read both in the source code of one widely used open-source embedding library, version 6.1.0.

• Long text is cut, not refused. Its text tokenizer settings turn truncation on by default, so words past the model's limit simply do not count. Split long documents yourself.
• Scaling is off by default. Its encode call leaves its normalisation option off (False) unless you turn it on, though some models include a scaling step themselves. Use the length check above, then pick cosine or dot product.

Both are as read on 19 October 2026 in version 6.1.0. Defaults can change between versions, so pin your version and re-check after upgrades.

Choosing the model

To pick a model, start with a public benchmark (a shared set of tests that many models are scored on), then test on your own data. The 2022 benchmark paper above covers 8 kinds of tasks, 58 datasets and 112 languages.

Its authors found that no single embedding method was best at every task. So a high benchmark rank is a shortlist, not a decision.

The marketplace team takes three candidate models and runs 300 saved searches through each, every one paired with the listing that buyer bought. It keeps the one that puts the right listing in the top 10 most often.

Explain them as one tool with a clear job: turning text into vectors so you can find items by meaning. Then show where that job ends and plain code takes over.

1. 1

   Name the problem

   Keyword search misses synonyms, other phrasings and other languages. Give one concrete miss, like Bombay and Mumbai.

2. 2

   Define the embedding

   One fixed-length vector per item, from an embedding model. The same model embeds the items and the searches.

3. 3

   Pick the similarity

   Cosine similarity, or the dot product once vectors are scaled to length 1, which ranks the same. Set any threshold from labelled pairs, not a guess.

4. 4

   Size it

   Items times dimensions times 4 bytes for 32-bit numbers. Say whether a full scan is fast enough, or whether you need a vector index.

5. 5

   Plan updates

   A background worker embeds new and edited items. Every vector stores its model version, and a model change is a planned full re-embed.

6. 6

   Name the limits

   Negation, numbers and codes. Handle them with filters and exact matching, and measure search quality on saved real searches.