Tokens: how AI reads your words

In 1985, Arjun’s grandfather sent a telegram home from Kanpur: “REACHED SAFELY STOP SEND MONEY STOP”. Every word cost money, so nobody wrote “Dear Amma, I hope this finds you well.”

Forty years later, Arjun builds a small AI helper for his college fest. The bill arrives in a strange unit. Not words. Not letters. Tokens. By the end of this lesson, you’ll know exactly what that unit is, and why it explains some of AI’s oddest habits.

In module 1 you saw that a model learns patterns from huge numbers of examples. But a computer cannot learn from letters on a screen. It only does maths. So before any learning or answering, your text has to become numbers.

The first step is cutting the text into pieces. The pieces are not always whole words, and they are not single letters either. They sit somewhere in between.

A short, common word like “the” or “love” is usually one token. A long or rare word like “unbelievable” may be cut into pieces such as “un”, “believ” and “able”. An exclamation mark is usually a token of its own.

Every token in the list has an ID number, the way every train in India has its own five-digit number. The tokenizer looks up each piece and swaps it for its ID. From then on, the model sees only a row of numbers.

Modern models use vocabularies of roughly thirty thousand to a few hundred thousand tokens. The list is chosen once, before training, and then it stays fixed for that model.

How is the list chosen? A popular method starts with tiny pieces, like single characters. It then scans a mountain of text and glues together the pair that appears most often. “t” and “h” become “th”. Later, “th” and “e” become “the”. After many thousands of merges, common words are single pieces and rare words stay in parts.

That is Ramesh’s order book again. Frequent orders earned a mark. Frequent letter pairs earned a token. And because the smallest pieces are always in the list, any text at all can still be written down, even a word nobody has seen before.

The box below starts with “I love masala chai! ☕”. Each coloured chip is one token. A small ␣ mark shows where a token carries a space. Under the box you’ll see how many tokens there are and how many characters each token holds on average. You’ll also see a rough rupee cost for a million such messages, at an example price.

• Tap English. Each common word is one token, the “!” is its own token, and the cup emoji takes two.
• Tap Hindi. A sentence of similar meaning needs almost twice as many tokens here.
• Tap “Long words”. Watch a long word like “internationalization” break into several pieces.
• Tap Code. Symbols like “=” and “*” become tokens of their own.
• Tap Emoji. One small picture often takes more than one token.
• Turn on “Show as numbers” to see the row of IDs the model actually receives.
• Type your own name. Common names may be one token; unusual spellings break into pieces.

Here is a handy rule of thumb for English. One token is roughly four characters, or about three quarters of a word. So a hundred tokens is roughly seventy-five words, and a one-page essay of five hundred words is roughly six or seven hundred tokens.

Tokens are AI’s meter, like the meter in an auto-rickshaw. Companies that offer AI models charge a price per million tokens. They count the tokens you send in and the tokens that come back, and output tokens usually cost more than input tokens.

Prices vary a lot. Depending on the model, a million tokens can cost anywhere from a few rupees to a few thousand rupees. One message is tiny, but an app with thousands of users sends millions of tokens a day.

Tokens also set the speed. As you’ll see in the next lesson, the model writes one token at a time. A longer answer means more tokens, and more tokens mean more waiting. And every model can only look at a limited number of tokens at once. You’ll meet that limit, the context window, later in this module.

Why do Indian languages often cost more? A tokenizer learns its pieces from text, and most of that text has been English. Hindi and Tamil words were seen less often, so fewer of them earned a single token. Newer tokenizers have narrowed the gap a lot, but a Hindi sentence still often needs more tokens than its English twin.

Ask an AI how many times the letter “r” appears in “strawberry”, and for a long time many models said two. Why would a clever system miss something a child can count?

Because the model never saw the letters. It saw something like “str”, “aw” and “berry” as three ID numbers. Asking it to count letters is like asking Ramesh bhaiya how many letters are in his mark “AD”. He knows it means adrak chai, but the mark itself does not show the spelling.

The same root cause trips models on rhymes, reversing a word, and exact sums with long numbers. Newer models handle many of these better, often by working step by step or by using a calculator tool. The underlying reason stays the same: the model thinks in tokens.