LLM Tokenization Explained: Why Hindi Costs More Tokens

The same work costs more, fits less and runs slower in Hindi, because the model counts Hindi text in many more tokens than English. LLM tokenization is the step that decides that count, and you can measure it before you pay.

Picture a common setup. A Hindi news app sends about 1,000 articles every morning to a model provider, a company that runs a large AI model and charges per use. The model writes a 60-word Hindi summary of each. The setup is made up, but the problem is real.

After the first month, the bill is about Rs 73,800. A similar English summariser, handling the same number of articles, pays about Rs 26,100. Same kind of model, same kind of work, nearly three times the bill.

The team checks one story in both languages. The English version has 400 words and the Hindi version has 410. Yet the provider reports 550 input tokens (tokens in the text you send) for English and 1,500 for Hindi. Published studies have measured gaps this size and larger, as a later section shows.

You need five words: token, tokenizer, vocabulary, token ID and context window. Two more appear everywhere. A model, or LLM (large language model), is the AI program that reads and writes text. A prompt is the text you send it, such as the team's instructions plus one article.

Token. A token is one small piece of text that the model reads or writes as a single unit. It can be a whole word, part of a word, a punctuation mark, or even part of one letter. Example: a common English word is often one token, while a Hindi word may take several.

Tokenizer. A tokenizer is the program that cuts text into tokens before it reaches the model, and joins tokens back into text afterwards. Each model family usually comes with its own tokenizer. So the same text can give different counts on different models.

Vocabulary. The vocabulary is the fixed list of every token a tokenizer knows, set when the tokenizer is built. A 2024 study calls 32,000 entries the conventional size and argues that larger models deserve larger vocabularies. If a whole Hindi word is not on the list, it is built from smaller pieces that are.

Token ID. A token ID is the number that stands for one token in the vocabulary. The model never sees letters, only these numbers. Example: in the small tokenizer built later in this guide, roads with a space in front is token 296.

Context window. The context window is the most tokens a model can handle in one request. In this guide's scenario it must hold the prompt and the reply together, so check how your model counts it. Example: with an 8,000-token window, the news app fits about five Hindi articles in one prompt, but about thirteen English ones, leaving room for the reply.

LLM tokenization is the step that cuts your text into tokens, the units a model reads and writes. Each token matches one ID in the vocabulary, so your prompt becomes a list of numbers. The reply comes back as more numbers, which the tokenizer turns back into text.

Whole words would make a poor unit, because there are far too many of them. New names, typos, hashtags and words from hundreds of languages would never fit in one list. Single letters would be poor too, because every sentence would become very long.

Tokens sit in the middle. Common words get one token each, and rare words are built from a few smaller pieces. In a byte-level tokenizer, anything at all, even brand-new Hindi slang, can still be written from the smallest pieces of all: single bytes, explained next.

A byte is the basic unit computers use to store text, a number from 0 to 255. Computers usually store text as bytes using a standard called UTF-8. In UTF-8, an English letter takes one byte, and each Devanagari letter, the script used to write Hindi, takes three.

You can check this in Python. The line print(len("खबर".encode("utf-8"))) prints 9, because खबर (khabar, which means news) has three letters of three bytes each. The same word typed in English letters, khabar, is 6 bytes.

Here is the path one article takes before the model sees it. Notice that the bill and the limits are counted only at the last step.

This is how byte-level tokenizers work. Others start from whole characters instead of bytes. Either way, known groups become one token, and everything else is built from smaller pieces.

Byte-pair encoding, or BPE, builds a vocabulary by starting with single bytes or letters. It then keeps gluing together the pair that appears most often in a large pile of training text. Each glue step is called a merge, and the ordered list of merges becomes the tokenizer.

Think of a printing press that sets wedding cards with metal type. The printer keeps ready-made blocks for phrases he sets every week, like Shubh Vivah. Placing one block is one move, while a rare family name is set letter by letter.

Map it part by part. Each type block is a token, the shelf of blocks is the vocabulary, and the weekly orders are the training text. Phrases ordered often get their own block, just as frequent pairs get merged.

Where the picture breaks: a printer can cast a new block whenever he likes. A tokenizer's list is frozen once it is built, so it never learns that your app now sends mostly Hindi.

BPE began in 1994 as a data compression method, a way to make files smaller. Sennrich and colleagues adapted it in 2015 so translation systems could handle rare words, merging letters instead of bytes. A 2019 paper by Wang and colleagues ran it over raw UTF-8 bytes, so no text is ever unknown to the tokenizer.

A tiny BPE example by hand

Say the training text is a tiny pile of English words. The word news appears 5 times, new 3 times and newer 2 times. Start with single letters, then follow the merges below.

1. 1

   Count the pairs

   Count every pair of neighbouring letters. The pair n and e appears 10 times, once in every word. The pair e and w also appears 10 times. When two pairs tie, a fixed rule picks one; here the pair seen first wins.

2. 2

   Merge 1

   Glue n and e into one new symbol, ne. Now news is written as ne, w, s, and new is written as ne, w.

3. 3

   Merge 2

   Count again. The pair ne and w appears 10 times, more than any other, so glue it into new. Now new is a single token everywhere.

4. 4

   Merge 3

   Count again. The pair new and s appears 5 times, which is now the top count. Glue it, and news becomes a single token too.

5. 5

   Stop at the chosen size

   Repeat until the vocabulary reaches the size the builder chose. In the original method, the number of merges is the only setting.

Now the tokenizer meets a word it never saw, like newsroom. It does not fail. It uses its merges to get news, then falls back to single letters for r, o, o and m. That is five tokens for one word: rare text costs more, but nothing is impossible to write.

What just happened? Nothing in the method knows what a word means. It only counts, so whatever was common in the training text wins the short, cheap tokens. The loop below is the whole training method.

Hindi needs more tokens when a tokenizer learned few merges for it, which happens when Hindi was a small share of its training text. With fewer ready-made pieces, each Hindi word is built from many small ones.

Go back to the printing press. Shubh Vivah has its own block because the printer sets it every week. For a tokenizer trained mostly on English, English is the Shubh Vivah, and Hindi is the rare family name, set piece by piece.

The script widens the gap. Each Devanagari letter is three bytes, against one for an English letter. So खबर is nine bytes, and depending on the merges a byte-level tokenizer learned, it might be one token or as many as nine.

Vowel signs add more. A syllable like कि (ki) looks like one shape, but underneath it is two characters: the letter क plus a vowel sign. That is six bytes. Joined letters work the same way, so a short Hindi word can hide many bytes.

Researchers have measured this. A 2023 study by Petrov and colleagues found the same text can need up to 15 times more tokens in one language than in another. The gaps remained even in tokenizers trained on purpose for many languages.

Another 2023 study, by Ahia and colleagues, found that, on the commercial model service they tested, speakers of many supported languages were overcharged while getting poorer results. So the problem is not only about Hindi. Measure your own languages rather than assume.

The chain below shows how one fact about training text ends up on the news app's bill. A later section builds a tiny tokenizer so you can watch the first link happen on your own machine.

Here is the scenario's story side by side. Say the prices are Rs 1 per 1,000 input tokens and Rs 4 per 1,000 output tokens, the tokens the model writes. To keep the maths simple, the table counts the article and the summary only.

Tokens drive all three. You pay per token, the context window is counted in tokens, and the model writes its reply one token at a time. So text that needs about three times the tokens costs more, fills the window faster and takes longer.

Cost. Model providers usually charge per token, and input and output tokens can have different prices. With the scenario's prices, one Hindi article costs Rs 2.46 against Rs 0.87 in English. Over 1,000 articles a day for 30 days, that is Rs 73,800 against Rs 26,100.

Context. The team wants to send five related articles together, for one combined summary of a big story. In English that fits easily. In Hindi, five articles take about 7,500 of the 8,000 tokens, leaving almost no room for instructions or the reply.

Speed. A model reads the whole prompt in one parallel pass, then writes the reply one token at a time, each step using the token before it. In standard decoding, which writes one token per step, a reply with three times the tokens takes three times as many steps. A longer prompt adds time too, but it is read in parallel.

Engineers measure these two waits as time to first token and time between tokens. A longer Hindi prompt raises the first. More reply tokens mean more gaps between tokens, so the full reply takes longer.

You count tokens by running the exact text you will send through the same tokenizer your model uses. Check your provider's documentation for how it supports this, such as a tokenizer you can run yourself or a counting request.

Counting with the wrong tokenizer gives wrong numbers, because each tokenizer has its own vocabulary and merges. Compare your counts with the usage numbers your provider reports for real requests, and fix the gap before you trust a budget.

Count the final prompt, not its parts. Merges can cross the point where you join two strings, so the counts of the parts do not always add up. A later section shows a case where 1 token plus 1 token makes 1.

Here is a check to run before every request. You pass in count_tokens, a function that wraps your model's own tokenizer and returns how many tokens a text makes.

PRICE_IN = 1 / 1000     # Rs 1 per 1,000 input tokens (made up)
PRICE_OUT = 4 / 1000    # Rs 4 per 1,000 output tokens (made up)
CONTEXT_LIMIT = 8000    # context window, in tokens
MAX_SUMMARY = 300       # most tokens the model may write
MARGIN = 50             # room for formatting tokens around the prompt

def plan_request(prompt, count_tokens):
    tokens_in = count_tokens(prompt)    # count the final prompt
    if tokens_in + MAX_SUMMARY + MARGIN > CONTEXT_LIMIT:
        return None                     # too long: split the article first
    worst_cost = tokens_in * PRICE_IN + MAX_SUMMARY * PRICE_OUT
    return tokens_in, round(worst_cost, 2)

The if statement checks the window before any money is spent: the prompt, the longest allowed reply and a margin must all fit. Send MAX_SUMMARY to the provider as the output limit too, so the worst-case cost really is the worst case.

Chat requests can add formatting tokens around your text, which is what MARGIN is for. Size it from the gap between your count and the usage the provider reports.

To check the function, pass a stand-in counter. plan_request("text", lambda t: 1550) returns (1550, 2.75), the scenario's 1,500-token article plus 50 for instructions. A count of 7,700 returns None, because 7,700 plus 300 plus 50 is more than 8,000.

Here is the full check for every article. Follow the No branch to see what happens to a very long story.

Then measure a planning number you can trust: tokens per word on a sample of your own articles in each language. Divide the total token count by the total word count, and budget Hindi with the Hindi figure.

All three build a vocabulary of subword pieces, parts of words, from training text. They differ in how they choose pieces: BPE merges by raw frequency, WordPiece merges by likelihood, and Unigram starts big and prunes.

Likelihood here is a score of how well the vocabulary explains the training text, where higher is better. A 2018 paper by Kudo describes the Unigram method, and notes that WordPiece uses likelihood instead of frequency.

Unigram has two extras. It always keeps every single character seen in training, so any text made of those characters can be written. When it splits text, it can pick the most likely split or sample a different one, which the paper used during training to improve translation models.

For cost work, the method matters less than the vocabulary it ended with. Whichever method your model's tokenizer uses, measure tokens per word on your own text.

You can write a working byte-level BPE tokenizer in about 45 lines of plain Python. It will not match any real model's counts, but it shows the mechanism behind them: the training mix decides who gets cheap tokens.

import re
from collections import Counter


def words(text):
    # each word keeps the space in front of it
    return re.findall(r" ?\S+", text)


def merge(word, pair, new_id):
    out, i = [], 0
    while i < len(word):
        if word[i:i + 2] == pair:
            out.append(new_id)
            i += 2
        else:
            out.append(word[i])
            i += 1
    return tuple(out)


def train(text, num_merges):
    counts = Counter(tuple(w.encode("utf-8")) for w in words(text))
    merges = {}  # (left ID, right ID) -> new ID, in learned order
    for new_id in range(256, 256 + num_merges):
        pairs = Counter()
        for word, n in counts.items():
            for pair in zip(word, word[1:]):
                pairs[pair] += n
        if not pairs:
            break
        best = max(pairs, key=pairs.get)  # ties: first pair seen wins
        merges[best] = new_id
        counts = Counter({merge(w, best, new_id): n for w, n in counts.items()})
    return merges


def encode(text, merges):
    ids = []
    for w in words(text):
        word = tuple(w.encode("utf-8"))
        for pair, new_id in merges.items():
            word = merge(word, pair, new_id)
        ids += word
    return ids

The words function splits text into words, each keeping the space in front of it. The train function counts neighbouring pairs inside each word, merges the top pair into a new ID from 256 upwards, and repeats. The encode function replays the merges in learned order on new text.

Now train it twice with 100 merges. Both runs use the same English text ten times. The first run adds one copy of the matching Hindi text, and the second adds ten copies.

ENGLISH = ("Heavy rain closed many roads in the city. "
           "The city council approved a new budget for roads and schools. "
           "Trains were late because of thick fog. "
           "The state will open new schools next year. ")
HINDI = ("भारी बारिश से शहर की कई सड़कें बंद हो गईं। "
         "नगर परिषद ने सड़कों और स्कूलों के लिए नया बजट मंज़ूर किया। "
         "घने कोहरे के कारण रेलें देर से चलीं। "
         "राज्य अगले साल नए स्कूल खोलेगा। ")
TEST_EN = "Thick fog closed schools and roads in the city."
TEST_HI = "घने कोहरे से शहर में स्कूल और सड़कें बंद रहीं।"

for hindi_copies in (1, 10):
    merges = train(ENGLISH * 10 + HINDI * hindi_copies, 100)
    en, hi = encode(TEST_EN, merges), encode(TEST_HI, merges)
    print(hindi_copies, len(en), len(hi), round(len(hi) / len(TEST_HI.split()), 1))

Pause and predict: when Hindi gets an equal share, what happens to the Hindi count, and what happens to the English one?

On Python 3.13, the script prints 1 13 69 6.9 and then 10 16 31 3.1. The Hindi test sentence drops from 69 tokens to 31, or from 6.9 to 3.1 tokens per word. The English sentence rises from 13 tokens to 16.

What just happened? The vocabulary has a fixed budget of merges. Giving Hindi a fair share bought it ready-made pieces, and those merges no longer went to English. Any BPE tokenizer with a fixed vocabulary size faces the same trade.

Keep the scope in mind. This is 100 merges on four sentences per language, so the exact numbers say nothing about real models. It also drops extra spaces and line breaks, which real tokenizers keep. The direction is the point, and you can rerun it on your own text.

Numbers, spaces and code tokenize strangely because tokenizers split them by learned patterns and fixed rules, not by meaning. The same value or word can turn into different tokens depending on what surrounds it.

Numbers. Some tokenizers give every digit its own token. Others have tokens for numbers of one to three digits and cut long numbers into groups of three from the left, so 12345 becomes 123 and 45. Those groups do not line up with thousands and hundreds.

A 2024 study by Singh and Strouse tested addition with both kinds of grouping. Accuracy was up to about 20 percentage points higher when commas forced the groups to line up from the right. The authors report a smaller gap in the larger model they tested, and suggest scale may help.

Indian number writing adds a twist. Under left-to-right grouping, 123,456 splits as 123, comma, 456. Written the Indian way as 1,23,456 (1.23 lakh, or 123,456), it splits as 1, comma, 23, comma, 456. Same value, different tokens.

So keep arithmetic in plain code. Let the model write the words around the numbers, and check every number in the output against the source.

The tiny tokenizer from the last section, trained with ten copies of Hindi, shows more quirks:

• Spaces join the word after them. Roads with a space in front is one token, ID 296. The same word at the very start of the text, with no space, is five.
• Capital letters change tokens. Roads and ROADS are five tokens each, because the training text never had them in those forms.
• Counts do not add up. Road with a space in front is 1 token and s is 1 token, yet joined they make 1 token. That is why you count the final prompt.
• Formatting costs tokens. A byte-level tokenizer keeps every space and line break as text, so deep indentation and blank lines are counted and billed.
• JSON, a common text format for data, repeats its key names in every record. Sending 100 records repeats every key name 100 times.

Streaming breaks. When a byte-level tokenizer splits a Devanagari letter across tokens, one token alone is not valid text. If you decode and show each token as it arrives, readers see broken characters.

This is common for rare scripts. In the tiny tokenizer trained with one copy of Hindi, 64 of the 69 Hindi tokens are not whole characters on their own. The 2019 byte-level paper notes the same: its pieces can be partial characters.

The fix is to hold incomplete bytes until a character is whole. Python's incremental UTF-8 decoder does exactly that.

import codecs

data = "खबर".encode("utf-8")                         # 3 letters, 9 bytes
print(data[:2].decode("utf-8", errors="replace"))    # half a letter

decoder = codecs.getincrementaldecoder("utf-8")()
for i in range(0, len(data), 2):                      # pretend each token is 2 bytes
    print(repr(decoder.decode(data[i:i + 2])))

The first print shows the replacement mark �, which is what a careless stream shows. The loop prints '', 'ख', 'ब', '' and 'र': each letter appears only once its three bytes have arrived.

If you write your own stream reader that turns bytes into text, buffer like this. Test it with Hindi before you ship, because English text rarely shows the bug.

You reduce Hindi token counts by sending less text, choosing a model whose tokenizer packs Hindi well, and capping how much the model writes. You cannot change how a tokenizer cuts a word, but you control what you send and what you ask for.

Here is what the team tries, in order of effort:

• Clean the input. Remove bylines, repeated menus, advert text and leftover web page code before sending. In this scenario that cuts 15 percent of input tokens (made up).
• Cap the output. Ask for a fixed length, such as 60 words, and set an output token limit so the reply cannot run on.
• Send only what the task needs. For a one-line headline, the first few paragraphs may be enough instead of the full article.
• Test English instructions. The fixed instructions can be in English while the article and summary stay in Hindi. Count both versions, and check that summary quality holds before you switch.
• Compare tokenizers when choosing a model. Count the same 200 Hindi articles with each option. A model that packs Hindi well can be cheaper for you, even at a higher price per token.

Cleaning the input alone brings the Hindi cost from Rs 2.46 to about Rs 2.24 per article. The gap with English shrinks but does not vanish, so the team adds tokens per language to its monthly report.

Link tokens to cost, limits and speed, then show that you would measure tokens on real data for every language the product serves.

1. 1

   Define a token

   Say a token is a piece of text with its own number, and that providers bill and models limit text in tokens.

2. 2

   Explain how tokenizers are built

   Describe BPE in one line: start from bytes or letters and keep merging the most frequent pair until the vocabulary reaches its chosen size.

3. 3

   Name the language gap

   Point out that languages rare in the training text, like Hindi, need more tokens for the same meaning.

4. 4

   Do the maths out loud

   Estimate tokens per request for each language, like the scenario's 1,500 against 550. Turn that into cost per day and window use, and say the numbers are guesses to check.

5. 5

   Add a check before sending

   Count the final prompt with the model's own tokenizer, check the window, cap the output and log the real usage the provider reports.

6. 6

   Mention the quirks

   Say that numbers and code split oddly, so you keep arithmetic in normal code. Add that streamed Hindi needs byte buffering.