Voice Assistant System Design: Speech In, Speech Out

Callers hang up. Silence on a phone line sounds like a dead line, and a slow voice bot is mostly silence. Good voice assistant system design is largely the work of removing that silence.

Picture a common setup. A farmers' helpline near Bhopal takes calls in Hindi about crop prices, rain, seeds and pests. The setup is made up, but the problem is real.

Most price questions are about the mandi, the local wholesale market where farmers sell their crop. In the kharif season, the monsoon sowing months, 45 human advisers take about 4,000 calls a day. In the busiest weeks a farmer waits 11 minutes on hold.

About 60 percent of calls ask the same few things: today's soybean price, the rain forecast, which seed to buy. So the team wants a voice assistant that answers those at any hour. The advisers can then spend their time on sick crops and hard cases.

The first version takes two weeks to build. It records the farmer's whole question, turns it into text, asks the model for a full reply, turns that into speech, and only then plays it. In testing, farmers wait about 6.5 seconds after they stop talking.

The pilot shows four problems:

• Farmers say hello again into the silence, and that extra hello becomes part of their next question.
• Some hang up, sure the line has dropped.
• When a farmer tries to correct the bot mid-reply, it keeps talking over them.
• A price like Rs 4,350 per quintal (100 kg) comes out as a jumble of digits read in English.

Each problem maps to one part of the design below: speed, turn-taking, interruptions and how numbers are spoken.

You will meet six terms in every voice design. Here they are with helpline examples.

Speech-to-text. STT, also called speech recognition, turns audio into written words. A farmer asks for the soybean price in Hindi, and STT returns the words "soybean ka bhav kya hai".

Language model. A language model is software that reads text and writes text back. Here it reads the farmer's words plus facts from the price database, then writes a short Hindi reply.

Text-to-speech. TTS turns written text into spoken audio, which for the helpline means a calm Hindi voice.

Voice activity detection. VAD is a small, fast check that says, for each short slice of audio, whether someone is speaking. It cannot tell what was said, only that there was speech.

Endpointing. Endpointing is the decision that the caller has finished their turn. It uses VAD's silences, and good systems also look at the words.

Barge-in. Barge-in is when the caller starts talking while the bot is still speaking. A good bot stops, listens and answers the new words.

Voice assistant system design is the plan for how a caller's voice travels to a model and comes back as speech, fast enough to feel like a conversation. It covers the parts, the order they run in, the time each may take, and what happens when one fails.

Most voice assistants are built as a pipeline, a chain of stages where each stage's output feeds the next: STT, then the model, then TTS. Some newer systems use one model that takes audio in and gives audio out.

A 2024 paper on one such model, which listens and speaks at the same time, reports a delay of about 200 ms in practice. Its authors argue that pipelines add seconds of delay, lose emotion in the voice and ignore people talking over each other.

The helpline still picks the pipeline. It needs text it can log, check against approved advice and search for every call. A pipeline also lets the team choose the STT and TTS that handle Hindi best.

Here are the parts. The phone gateway is the service that connects phone calls to your own software. Follow the call from the farmer's phone round the loop and back to the gateway.

The call orchestrator holds the call state, meaning everything said so far, and runs every other stage.

There is no standard number, so set a target and measure it. The helpline aims for under 2 seconds from the farmer's last word to the bot's first sound, and even that is slow next to people talking.

A 2015 review of turn-taking research reports that gaps between turns in conversation average around 200 ms. Yet preparing even one new word takes people around 600 ms or more. So the authors conclude that listeners plan their reply while the other person is still talking.

The phone network adds its own delay. ITU-T G.114 (2003, in force as of 19 October 2026), the telecom standard on one-way delay, recommends not going over 400 ms one way for general network planning. It adds that most uses are barely affected below 150 ms, while some highly interactive speech may be affected even below 100 ms.

Latency means the time one step takes. A latency budget splits the total wait into a share for each stage. With a budget, a slow call points straight at the stage that went over. Here is the first version next to the streaming version.

In the first version, the model and TTS took 3.9 seconds between them, because each waited for the whole reply. The fixed 1.2-second silence wait came next.

Streaming makes a voice assistant faster because every stage starts on the first piece of its input instead of waiting for all of it. The stages then overlap in time instead of running one after another.

Think of a message relayed by walkie-talkie across a big farm. A worker in the far field radios the pump house, and the pump house radios the farmhouse. A good relay passes on each short line as soon as it hears it. A poor relay waits for the whole message, writes it down, and only then starts.

• The worker in the field is the caller.
• The pump house is STT and the model, turning what it hears into a reply.
• The farmhouse is TTS and the phone gateway, where the reply is spoken.
• Each short line is one chunk of words or audio.

Where the picture breaks: a relay passes on the same words, but each stage here changes their form, from audio to text and back to audio. And STT can still change a word after passing it on, which no radio relay does.

The first version was the poor relay. Here is the good relay, stage by stage.

• STT streams partial results: its best guess at the words so far, updated as more audio arrives. When the farmer stops, only the last word or two still needs work.
• The model streams its reply in small pieces of text as it writes them.
• TTS turns the first sentence into audio while the model is still writing the second.
• The phone gateway plays each chunk of audio as soon as it arrives.

Partial STT results can change. A first guess might read "soya been ka bhav", and a moment later it becomes "soybean ka bhav". So the orchestrator may start cheap work early, like guessing the crop, but sends the model only the final words.

The step in the middle, sending each finished sentence to TTS at once, sounds trivial. The engineer section below shows the obvious code for it and the bug that code hides.

A voice assistant decides the caller has finished when VAD hears enough silence, and good systems also check whether the words so far sound complete. This is endpointing, and it is one of the hardest timing calls in the pipeline.

People on walkie-talkies solve this with one word: saying over hands the turn to the other side. Callers on a phone never say over. They pause to think, to recall a number, or to let a tractor pass.

So the bot has to guess. If it waits too little, it cuts farmers off mid-sentence. If it waits too long, every reply feels slow. The first version waited for 1.2 seconds of silence every time.

The fix reads the words as well as the silence. If the last word is aur, ya or lekin, meaning and, or, but, the farmer is probably not done. Then the bot waits 1.2 seconds; otherwise it waits 600 ms. Both limits are the team's choices (made up).

Research goes further than fixed rules. The authors of a 2022 turn-taking method note that silences inside a speaker's turn are often longer than silences between turns.

Their model, trained on recorded telephone conversations, predicts the voice activity of both speakers over the next 2 seconds. From that it can anticipate turn shifts and backchannels, short sounds like haan or hmm that mean carry on.

When the caller interrupts, the bot should stop speaking quickly, drop the rest of its reply and listen. A walkie-talkie cannot do this: while one side holds the talk button, the other side cannot be heard.

The first version behaved like a walkie-talkie and stopped listening while it spoke. A phone line carries both voices at the same time, so the bot can keep listening, and it should.

Not every sound is an interruption, though. Some of the bot's own voice can come back down the line, for example from a handset on speakerphone. An echo canceller removes most of it.

The canceller takes the audio the bot just played as a reference and subtracts that audio's echo from what comes back. The manual for Speex, an open-source audio library with an echo canceller, warns that non-linear distortion cannot be modelled by its filter.

A cheap speaker driven into clipping distorts sound in exactly that way, so some echo can slip through. That is why the helpline also ignores speech shorter than 300 ms. Here is the whole check.

A phone call is hard for speech recognition because the audio is thin, often noisy, and full of mixed languages.

Thin audio on the phone line

A codec is a method for packing audio into bits. The classic telephone codec, ITU-T G.711, takes 8,000 samples a second with 8 bits each. Its two forms are mu-law and A-law, two ways of squeezing each sample into 8 bits. RFC 3551, an internet standard for live audio, gives them fixed codes, PCMU and PCMA, with an 8,000 Hz clock.

Sampling 8,000 times a second can only capture sounds below 4,000 Hz, half the sampling rate. Anything higher cannot be carried, so a converter filters it out first. The engineer section shows a 5,000 Hz tone almost vanishing when squeezed into phone format.

So test STT on 8 kHz audio, never only on clean recordings. A 2024 speech dataset covering 22 Indian languages collected part of its data over an 8 kHz telephone channel. The authors did this because many low-income users without smartphones reach automated phone menus over such lines.

Then add the real world: wind in an open field, a running pump, a crowded mandi, a handset held far from the mouth. The team keeps 500 real calls, saved with the callers' consent, as its test set. Our speech recognition system design guide covers how to measure STT on such a set.

Hindi, dialects and mixed languages

Farmers around Bhopal speak Hindi, but rarely the formal Hindi of government forms. They mix in local dialect words, and English words like urea, tractor and DAP, a common fertiliser. The authors of that dataset note that within one language, accents and everyday usage vary from region to region.

Numbers are the biggest risk on the helpline. A farmer says "saadhe chaar hazaar". That means four and a half thousand, so STT's output must become 4,500 before the price lookup.

The same problem runs the other way in TTS. The database says Rs 4,350 per quintal, and TTS must say it the way a farmer would. SSML, the W3C Speech Synthesis Markup Language (version 1.1, 2010), offers a say-as element for this.

Say-as tells the engine what kind of text something is, such as a number or a date. The standard does not list the allowed values, so each speech engine supports its own; check yours. SSML's lang element marks a word's language, which helps an engine that supports it say urea the English way.

Language also changes the model bill. Models read and write tokens, small pieces of text that are often a word or part of a word. A 2023 study found that the same text in different languages can need up to 15 times as many tokens in some cases. Count tokens on your own Hindi prompts before you budget.

A voice assistant needs a fallback for each way a call can fail: the bot mishears, the model cannot answer, or a whole stage goes down. The helpline's rule is simple: never leave a farmer in silence, and never guess.

The team adds one hard rule. Pesticide doses come only from approved advisories in the database. If no advisory matches, the call goes to a human adviser.

A confidence score is a number STT returns with its text, saying how sure it is. DTMF tones are the sounds a phone makes when you press its keys. Each key sends a fixed pair of tones, so there is no accent to misread.

Here is the ladder the team builds, from the cheapest fix to the most expensive.

Whole stages can fail too, so each gets its own fallback:

• Model slow or down: after 3 seconds with no first sentence, switch to fixed templates, like today's price read straight from the database.
• TTS down: play pre-recorded clips for the most common answers.
• STT down: switch the whole call to a keypad menu for prices and rain.
• Any slow moment: play a short recorded line, "ek minute, main dekh raha hoon" (one minute, I am checking), so the call is never silent.

A voice assistant's cost per call is the sum of four bills: phone minutes, STT minutes, model tokens and TTS output. Work it out for one call first, then multiply by calls per day.

Here is an average 3-minute call, with each service priced per unit (made up).

• Phone line: 3 minutes at Rs 0.40 a minute, so Rs 1.20.
• STT: 3 minutes of streamed audio at Rs 0.50 a minute, so Rs 1.50.
• Model: 6 turns at about Rs 0.15 each, so Rs 0.90.
• TTS: 1.5 minutes of bot speech at Rs 0.80 a minute, so Rs 1.20.

That is Rs 4.80 a call, or Rs 19,200 a day at 4,000 calls. The team then cuts two costs and stops a third from growing.

• Send only speech to STT, if your STT bills by audio minutes sent. VAD knows when the farmer is silent, so audio stops flowing in the gaps. Billed STT time drops from 3 minutes to about 1.3.
• Reuse fixed speech. The greeting, the menu and the consent notice are recorded once and replayed. TTS now bills about 1 minute a call.
• Keep prompts short. If your model API is stateless, meaning it remembers nothing between requests, each turn resends the call so far. So the orchestrator keeps a short summary instead of the full transcript. The model then stays at about Rs 0.90 a call instead of growing with every turn.

The new total is about Rs 3.55 a call: Rs 1.20 phone, Rs 0.65 STT, Rs 0.90 model and Rs 0.80 TTS. At 4,000 calls, that is Rs 14,200 a day. The Rs 5,000 daily saving comes to Rs 1.5 lakh (150,000) over a 30-day month.

One catch with sending only speech: VAD usually confirms speech a moment after it starts. Keep a little audio from just before it fires and send that too, or STT will miss the first sound of each turn.

Join the model's streamed pieces in a buffer. Whenever the buffer holds a complete sentence, send that sentence to TTS and keep the rest. The helpline's prompt asks the model to end Hindi sentences with a danda (।), the Hindi full stop.

The obvious version checks, after each piece, whether the buffer ends with a sentence mark: buffer.rstrip().endswith(("।", "?", "!")). It works whenever a sentence happens to end exactly where a piece ends, and nothing promises that it will.

Only at the very end. Its buffer never ends with a danda until the last piece, so the whole reply goes to TTS in one go. We ran that version on these four pieces: one TTS call, made after the last piece.

The fix looks for the last sentence mark anywhere in the buffer, not just at its end. This version runs on its own with Python 3, using a fake model stream so you can see the timing.

import re

SENTENCE_END = re.compile(r"[।?!]")    # the danda (।) is the Hindi full stop

def sentences(pieces, min_chars=20):
    buffer = ""
    for piece in pieces:                    # small bits of text, as written
        buffer += piece
        ends = [m.end() for m in SENTENCE_END.finditer(buffer)]
        if ends and ends[-1] >= min_chars:  # cut after the last full sentence
            yield buffer[:ends[-1]].strip()
            buffer = buffer[ends[-1]:]
    if buffer.strip():                      # whatever is left at the end
        yield buffer.strip()

reply = ["सोयाबीन का भाव", " 4,350 रुपये प्रति क्विंटल है",
         "। कल हल्की", " बारिश हो सकती है।"]

def model_stream():                         # stands in for the model
    for n, piece in enumerate(reply, 1):
        print("piece", n, "arrives")
        yield piece

for sentence in sentences(model_stream()):
    print("to TTS:", sentence)

Expected output:

piece 1 arrives
piece 2 arrives
piece 3 arrives
to TTS: सोयाबीन का भाव 4,350 रुपये प्रति क्विंटल है।
piece 4 arrives
to TTS: कल हल्की बारिश हो सकती है।

What just happened? Sentence one left for TTS as soon as piece 3 arrived, while the model was still writing. The decimal point in a price like Rs 4.50 is never a cut point, because only ।, ? and ! count. The 20-character minimum stops tiny fragments from becoming choppy separate clips.

In the orchestrator, each sentence goes to TTS and each audio chunk goes to the gateway. Run TTS in its own task fed by a queue, so reading the model's stream never waits for speech to be made. Also cap sentence length, so a model that forgets the danda still gets cut at a comma.

Count silence in 20 ms frames, reset the count whenever VAD hears speech, and end the turn when silence passes a limit that depends on the last word. This version returns the moment the turn ended, so you can test it with fake frames.

FRAME_MS = 20                            # VAD judges 20 ms of audio at a time
UNFINISHED = {"और", "या", "लेकिन"}       # aur, ya, lekin: and, or, but

def end_of_turn_ms(frames, is_speech, words_so_far):
    silence_ms, now_ms = 0, 0
    for frame in frames:
        now_ms += FRAME_MS
        if is_speech(frame):
            silence_ms = 0                   # any speech resets the count
            continue
        silence_ms += FRAME_MS
        words = words_so_far().split()
        limit = 1200 if words and words[-1] in UNFINISHED else 600
        if words and silence_ms >= limit:
            return now_ms                    # the turn ended here
    return None                              # audio ran out first

# 1 s of speech, an 800 ms pause, 1 s more speech, then 2 s of quiet
frames = ["speech"] * 50 + ["quiet"] * 40 + ["speech"] * 50 + ["quiet"] * 100
is_speech = lambda frame: frame == "speech"

print(end_of_turn_ms(frames, is_speech, lambda: "सोयाबीन का भाव और"))
print(end_of_turn_ms(frames, is_speech, lambda: "सोयाबीन का भाव"))

Expected output: 4000, then 1600, each on its own line.

What just happened? Both calls get the same audio: 1 second of speech, an 800 ms pause, 1 more second, then quiet. With aur at the end, the 800 ms pause is under the 1,200 ms limit, so the turn ends 1.2 s after the second part, at 4,000 ms.

Without aur, the 600 ms limit fires mid-pause at 1,600 ms, and the bot would cut the farmer off. A pause shorter than the limit never ends a turn, because speech resets the count.

Two details matter on real calls. STT's partial words trail the audio a little, so the rule may see aur a few frames late. The code also assumes STT writes Hindi in Devanagari; if yours writes Latin letters, add aur, ya and lekin too. And many VADs accept only fixed frame sizes, so check yours before choosing FRAME_MS.

Take the open-source WebRTC project's VAD, used from Python as webrtcvad-wheels 2.0.14.post1 (as of 19 October 2026). It accepts only 16-bit mono (one-channel) audio at 8, 16, 32 or 48 kHz. Its frames must be 10, 20 or 30 ms long.

We ran it: 20 ms frames at 8 kHz pass, and 25 ms frames raise an error. So decode G.711's 8-bit samples to 16-bit audio before VAD sees them.

Save a few seconds of call audio from your gateway and ask ffprobe what is inside. ffprobe is the free media inspector that comes with ffmpeg.

ffprobe -v error -show_entries stream=codec_name,sample_rate,channels \
  -of default=noprint_wrappers=1 call.wav

For a G.711 call saved as a WAV file, expect codec_name=pcm_mulaw (or pcm_alaw), sample_rate=8000 and channels=1, or 2 if both sides are in one file. If your gateway already decoded the call, codec_name=pcm_s16le (16-bit audio) is normal too. If you see 16000 or more, check where the audio came from; the tone test below shows why.

To hear what 8 kHz throws away, make a one-second 5,000 Hz tone at 16 kHz and squeeze it into phone format.

ffmpeg -y -v error -f lavfi -i "sine=frequency=5000:sample_rate=16000:duration=1" \
  -ar 8000 -c:a pcm_mulaw tone.wav
ffmpeg -i tone.wav -af volumedetect -f null - 2>&1 | grep mean_volume

We ran this with ffmpeg 6.1.1. Measured at 16 kHz before conversion, the tone's mean volume is -21.1 dB (decibels). After conversion it falls to about -62.9 dB. Lower decibels mean quieter, so that is close to silence. Change 5000 to 3000 and the mean volume stays at -21.1 dB.

A higher sample rate in a file proves nothing on its own. We resampled the phone-format tone to 48 kHz, and it stayed near silent at -65.0 dB.

Key presses may not reach you as sound at all. RFC 4733 lets a gateway send each key as a named event instead of audio. The reason it gives is that low-rate voice codecs cannot be guaranteed to carry the tones well enough for machines to recognise. So check whether your gateway delivers key presses as events or as audio before you build the keypad fallback.

Explain it as a pipeline with a time budget, then show how streaming, turn-taking and fallbacks keep it inside that budget.

1. 1

   Ask the questions that change the design

   Phone line or app? Which languages? How many calls at peak? What must the bot never get wrong?

2. 2

   Draw the pipeline with an orchestrator

   Phone gateway, VAD, streaming STT, model, TTS, and your orchestrator in the middle holding the call state.

3. 3

   Set a latency budget

   Pick a target for time to first audio, then give each stage a share. Say which shares you control and which you do not.

4. 4

   Stream every stage

   Partial STT results, streamed model output cut into sentences, and TTS audio played chunk by chunk.

5. 5

   Handle turns and barge-in

   Endpointing that uses silence plus words, and barge-in that cancels the audio and the unfinished reply.

6. 6

   Plan failures and cost

   Walk through the fallback ladder and cost per call. Then name the numbers you would watch: time to first audio, handoff rate and word error rate by region.