Voice and multimodal AI

Asha’s Nani in Lucknow has never typed a message in her life. But one evening she holds the phone close and says, “Raat nau baje BP ki dawai yaad dila dena.” Remind me about my BP medicine at nine tonight. The phone answers in a calm voice: “Done, Nani. I’ll remind you at 9 pm.”

No keyboard, no menus. For hundreds of millions of people, talking is far more natural than typing. So how does a model that works with tokens hear Nani, and talk back? Let’s follow her sentence.

A “mode” is a kind of information: words, pictures, sounds. Early chat apps only did text in and text out. By 2026, most big chat apps let you show a photo, speak a question, or share your screen, and many can answer by voice or with a picture.

The classic design is a relay race of separate parts. Each part does one job well and passes the baton on. Three of them are new words, so let’s meet them first.

In a normal phone call, people reply to each other within a fraction of a second. If an assistant goes quiet for two or three seconds, it feels broken. Every hop in the relay adds delay, so the waits pile up.

• Overlap the steps: transcribe while the person is still talking, and start speaking the first sentence before the whole reply is written.
• Keep replies short. A shorter answer means fewer tokens to write and less audio to make.
• Allow interruptions. If the person starts talking, stop speaking at once and listen. This is often called barge-in.
• Guess the end of a sentence carefully. Reply too early and you cut people off; too late and it feels slow.

Then there is how India speaks. Many people mix languages in one sentence, like Nani’s Hindi with “BP”. Accents change every few hundred kilometres. Numbers can come as “nau”, “nine” or “twenty-one hundred”. Speech-to-text has improved a lot for Indian languages, but mixed speech and noisy rooms still cause mistakes.

Since 2024, some assistants skip parts of the relay. A speech-to-speech model hears your tone, not only your words, and can answer with feeling and less delay. The trade-off: there is no clean text in the middle to check, log or search, so many products still use the relay, or a mix of both.

Remember tokens and embeddings? A model only works with numbers. So a photo is cut into a grid of small squares, called patches. Each patch is turned into an embedding, the same kind of meaning-numbers that words become.

Sound works in a similar way: audio is sliced into very short moments, and each slice becomes numbers. Once everything is numbers in one context window, attention can connect “vegetarian” in your question with the patch that says “paneer”.