Temperature: the creativity dial

Priya asks a chat app for a birthday wish for her dadi. She likes it, asks again, and gets a completely different wish. Same question, new answer. Meanwhile her friend Arjun is building a tool that reads bills, and he is annoyed because it wrote “₹1,240” once and “Rs 1240.00” the next time.

Both of them are meeting the same hidden setting. It has a funny name, temperature, and once you see what it does, AI answers stop feeling random.

You learned that a model writes one token at a time. It never knows the next word for sure. Instead, for every possible token it works out a chance. After “Happy birthday, dear”, the word “Dadi” might get a big share, “Nani” a smaller one, and “Mumbai” almost nothing.

Then the app has to choose one token from that menu. It could always take the top choice. Or it could roll a weighted dice, where likely words win more often but less likely words sometimes win too. That choosing step is called sampling.

Inside, the model first gives every token a raw score. Those scores are turned into chances that add up to 100%. Temperature steps in between: the scores are divided by the temperature before they become chances.

Divide by a small number, like 0.2, and the gaps between scores grow. The leader pulls far ahead, so the bars look sharp and tall. Divide by a big number, like 1.5, and the gaps shrink. The bars flatten, and words from the bottom of the list start getting real chances. At exactly 1, the model’s own chances are left as they are.

Remember that this happens for every single token. One surprising pick early on changes the text that follows, and the next menu changes with it. That is why a small bump in temperature can send a whole answer in a new direction.

Notice what you saw. At 0, the tallest bar takes 100% and “Let the AI pick” takes it every time; the label reads “Always the top word”. Near 1.5, the label says “Adventurous, sometimes odd”. The bars look much more alike, so each tap can give a different word, and the sentence can drift somewhere strange.

There is no best temperature, only the right one for the job. Ask yourself one question: do I want the same correct answer every time, or do I want variety?

So Arjun should set his bill reader near 0. Priya can leave her birthday wishes at a medium setting and enjoy the variety. Many apps allow a range of 0 to 1, others 0 to 2. Above about 1.2 or 1.5, most text starts to fall apart.

Think of a food delivery app showing you restaurants. One rule says “only show the top 5”. Another says “show the best ones until they cover 90% of the orders people usually place”. Both trim the long tail so you never see the dodgy places at the bottom.

Models have the same two trimming rules. Top-k keeps only the k most likely tokens, say the top 40, and picks among them. Top-p keeps the smallest group of top tokens whose chances add up to p, say 0.9, and picks among those. When the model is very sure, that group may hold one or two words. When it is unsure, the group grows.

Temperature changes the shape of the menu. Top-p and top-k cut its length. They are often used together. A common tip is to change one at a time, usually temperature, and leave the others at their defaults, so you know what made the difference.