Embeddings: meaning as a map

Asha has moved from Chennai to Delhi, and on her first Sunday she walks into a huge supermarket looking for curd. She has no idea which aisle it is in. Then she spots paneer, and right beside it: curd, butter and milk.

She did not need the exact shelf number. She only needed something close in kind, because the shop keeps similar things near each other. Hold on to that shop layout. It is exactly what an embedding is.

In the lesson on tokens, every piece of text became an ID number. But an ID is only a label. Think of roll numbers in a class register. Roll 41 and roll 42 sit next to each other, yet that tells you nothing about whether those two students are alike.

Token IDs work the same way. “cat” might be number 9,203 and “kitten” number 74,118. The numbers are far apart, even though the meanings are close. The model needs numbers where closeness actually means something.

Now imagine each word got only two numbers instead of hundreds. Two numbers are a point on a flat map: one for left and right, one for up and down. Put every word on that map, and something lovely happens. Fruits gather in one corner, vehicles in another, cricket words in a third.

The demo below is such a map, with about thirty words in groups: fruits, vehicles, animals, cricket, feelings and royalty. Tap any two words and a line joins them, with a percentage for how close in meaning they are.

• Tap two fruits. Then tap a fruit and a vehicle. Compare the two scores.
• Tap two words from the cricket group. They sit close because they appear in the same kind of sentences.
• Tap “bat” and “cricket bat”. Same spelling, different meanings, different spots: one “bat” is the animal that flies at night.
• Tap “king” and “queen”, then “king” and “prince”. All royalty, but each pair differs in its own way.
• Tap the “king − man + woman” button. Two arrows appear, man to woman and king to queen, and they point the same way.

Nobody types these numbers in. The model learns them from how words are used. A linguist, J. R. Firth, put the idea neatly in 1957: “You shall know a word by the company it keeps.”

Think of the sentence “I drank a cup of hot ___”. Chai, coffee and milk all fit that gap. Words that keep filling the same gaps get pushed closer together during training, through the same guess, check and nudge loop you met in module 1. Over billions of sentences, a map of meaning takes shape.

Directions on the map can carry meaning too. In a famous result from 2013 research on word embeddings, taking the spot for “king”, subtracting “man” and adding “woman” landed near “queen”. One direction had roughly learned “female”, another “royal”. It works roughly, not perfectly, so treat it as a lovely illustration rather than a law.

Rohan’s order from a shopping app arrives broken. On the help page he types “money back”. An old keyword search finds nothing, because the help article is titled “Refund policy” and shares none of his words.

Search with embeddings works differently. Every help article was turned into a vector ahead of time. Rohan’s question becomes a vector too, and the app looks for the stored vectors nearest to it. “Refund policy” comes out on top, because its meaning sits right next to “money back”.

Many real apps use both at once, which is called hybrid search. Here are some places you meet embeddings without noticing:

• Help pages and document search that understand what you meant, not only what you typed.
• Recommendations: songs or videos whose vectors sit near the ones you liked.
• Grouping thousands of customer complaints into a handful of themes.
• Spotting duplicate questions on a forum, even when they are worded differently.
• RAG, where an AI looks up your documents before answering. You’ll build that idea in module 4.

Embeddings also live inside every language model. The very first thing an LLM does with each token ID is look up its embedding. At that point “bank” has the same vector in “river bank” and “bank loan”. The next lesson shows how attention lets the neighbouring words reshape it.