Text to Speech System Design: Voices for a News App

The voice reads names, numbers and short forms wrongly, the first tap takes seconds to play, and the bill grows with every listener. None of these is mainly a model problem. Each one is a text to speech system design problem.

Picture a common setup. A regional news app in Assam publishes about 1,200 stories a day in Assamese, Bengali, Hindi and English, and around 400,000 people listen daily. The setup is made up, but the problems are real.

An average story is 350 words, which becomes about two and a half minutes of audio. People listen while cooking, driving or commuting, so the audio has to start fast and sound right.

The naive first version sent each story's raw text to a speech model every time someone pressed play. It looked simple, and it was wrong in four ways within a week.

• A flood story said "Rs 2.5 cr" as "R S two point five C R". Listeners could not tell it meant 2.5 crore (25 million) rupees.
• A cricket score of 245/6 came out as "two hundred and forty-five slash six", and local town names had the wrong sounds.
• During a cyclone alert, playback took 9 seconds to start, because every tap waited for the whole story to render (made up).
• The speech bill for the first month came to about Rs 6 crore (60 million rupees), far over plan (made up).

The rest of this guide rebuilds that app one decision at a time. Each fix answers one item on the list.

Text to speech system design is planning every part that turns written text into spoken audio and delivers it to listeners at scale. The model is only one part. Text cleanup, storage, delivery and quality checks matter just as much.

A few words first, each with a plain meaning.

• TTS, short for text to speech, is software that reads text aloud in a synthetic voice.
• Text normalization means rewriting text into the exact words a person would say: "NH-27" becomes "National Highway 27".
• A phoneme is one unit of sound in a language, like the "s" that starts Sualkuchi in English, a silk-weaving town about 35 km from Guwahati.
• A spectrogram is a picture of sound over time, showing which pitches are loud at each moment.
• A vocoder is the model that turns a spectrogram into the actual sound wave your speaker plays.

Many TTS systems turn text into audio in two model steps: an acoustic model predicts a spectrogram from the text, and a vocoder turns that spectrogram into a sound wave. A text front end prepares the input before either model runs.

This split is well established. A 2017 paper paired a network that predicts spectrograms from characters with a separate vocoder network that produces the waveform. On its own test set, listeners rated it 4.53 out of 5 on average, against 4.58 for professionally recorded speech. That average is a mean opinion score, explained later.

It is not the only design. A 2021 paper described a single network that goes straight from text to waveform, and reported more natural audio than the two-stage systems it compared against. The front end matters either way.

Here is the path one story takes in the two-stage design. Start at the story text and follow the arrows.

Each stage fails in its own way. A bad front end gives perfect audio of the wrong words. A weak acoustic model sounds flat or robotic. A weak vocoder adds buzz or a metallic edge.

Speed is rarely the bottleneck for stored news. A 2020 vocoder paper measured its full-quality version at 167.9 times faster than real time, at 22.05 kHz on one data-centre GPU. That covers the vocoder only; the acoustic model's time is extra.

Language choice is a design decision too. A 2022 study trained voices for 13 Indian languages and compared many setups. One model per language, trained jointly on a male and a female speaker, performed best.

So the app runs one fixed voice per language. Regular listeners come to trust a voice, and a sudden change sounds like a different channel.

You fix the text before the model sees it. Expand numbers, money and short forms into words, and give local names a stored pronunciation from a lexicon your team maintains.

A pronunciation lexicon is a list of words with exact pronunciations, written as phonemes. The W3C, the body behind web standards, publishes a format for it: the Pronunciation Lexicon Specification, or PLS, a Recommendation since 2008.

Each PLS entry pairs a written form, called a grapheme, with a pronunciation, called a phoneme. An alias element covers acronyms and short forms by giving other words to say instead. An alias can tell the engine to read "ASDMA" as its full name.

Telling the engine what kind of text it is

Some text means different things in different places. "245/6" is a cricket score in a sports story and a fraction in a maths story. "12/10" could be a date or a ratio.

SSML, the W3C Speech Synthesis Markup Language (version 1.1, 2010), lets you mark these cases inside the text with tags. The tags you will use most are these.

• say-as labels how to read a piece of text, such as a date or a phone number. SSML does not list the allowed labels, so check what your engine supports.
• phoneme gives an exact pronunciation for one word, for example in IPA, the international phonetic alphabet. The spec says engines should support IPA, not that they must.
• break adds a pause, either by strength, from none to x-strong, or by time, such as 300ms after a headline.
• prosody changes pitch, speaking rate or volume, for example a slower rate for election results.
• lang marks a switch of language, for an English quote inside a Hindi story.
• lexicon points the engine to an external pronunciation file, such as the newsroom's PLS file.

Build these tags in code from each story's section and language. Editors should never type SSML by hand.

For content that many people hear, like news, generate speech ahead of time and store it. Generate on demand only when the text is new for each listener, like a chatbot reply or a personal alert.

A film dubbing studio is a good picture of this. Here is how its parts map to the design.

• The dubbing director's cue sheet, listing how every name is said, is the pronunciation lexicon.
• Dubbing the film once before release is rendering each story once, when it is saved.
• Sending the same dubbed print to every cinema is copying the same audio file to every CDN server.
• Nobody dubs the film again for each ticket sold. The naive version did exactly that, once per play.

Where the picture breaks: a film is finished once, but news stories get corrected after release. Each corrected version needs its own new recording, which a later section handles with file names.

Pause and predict. The app has 1,600 story versions a day and about 2 million plays. How many times more rendering does the naive version do?

The answer is 1,250 times. Say one render costs about Rs 1 on the app's own GPU servers (made up). About 1,200 stories plus 400 edited versions means 1,600 renders, or Rs 1,600 a day.

Rendering on every play multiplies the work by the plays. With 2 million plays, the same audio costs Rs 20 lakh a day, or Rs 6 crore over 30 days. News is written once and heard many times, so pay for it once.

Breaking news uses a fourth option, shown later: render once, sentence by sentence, and share the chunks with every listener.

You store each rendered file in object storage and deliver it through a CDN, so listeners download audio from a server near them. The TTS workers run only when a story is new, its spoken words changed or the engine version changed.

Object storage is a cheap store for whole files, each addressed by a name called a key. A CDN, short for content delivery network, is a set of servers in many cities that keep copies of popular files.

The file format matters for phones on weak networks. Opus is an open audio codec, a method for compressing sound, defined by the IETF in RFC 6716. It scales from narrowband speech at 6 kbit/s up to stereo music at 510 kbit/s.

The RFC names 28 to 40 kbit/s as the sweet spot for fullband speech in 20 ms frames, so 32 kbit/s is a sensible choice. A 150-second story then takes 32 times 150, or 4,800 kilobits. That is about 600 kB (kilobytes), and a real encode lands close to it.

Here is the whole system. Notice that the listener never talks to the TTS workers directly.

The queue protects the workers. On election night, hundreds of stories change within minutes. The queue holds the jobs, and workers take them as fast as they can instead of falling over.

An engine without your rules guesses from the letters. You can watch one misread Indian news text with eSpeak NG, a long-standing open-source speech engine, in a few seconds.

On Ubuntu 24.04, install the espeak-ng package (version 1.51 there). Then run espeak-ng -v en-gb -q --ipa "Rs 2.5 cr". The -q flag stops it playing sound, and --ipa prints the sounds it would say in IPA.

The last row is the lesson. Once the text says exactly what to speak, even a simple engine gets it right. Notice also that it split 1,20,000 as if it were 1 and 20,000. Never assume an engine knows Indian digit grouping; test it like this.

The right sound also depends on the voice. An English voice should start Sualkuchi with s. In Assamese the name is said closer to "xualkusi", with x like the ch in Scottish loch.

So the lexicon needs one entry per language for the same name. Have a native speaker confirm each entry before it goes live.

SSML tags are just as easy to check. With the -m flag, espeak-ng reads SSML, and a say-as tag with interpret-as set to characters makes it spell ASDMA letter by letter.

You write rules that find each pattern, such as a rupee amount, and replace it with the words a person says. Short forms come from a list your newsroom keeps. Numbers go through a speller that knows lakh and crore.

Here is a small, complete normalizer for English stories. ALIASES holds short forms the newsroom uses often. MONEY finds "Rs" or the rupee sign, an amount with Indian-style commas, and an optional crore or lakh unit in any case, singular or plural.

import re

ALIASES = {"ASDMA": "Assam State Disaster Management Authority",
           "NH-27": "National Highway 27"}
UNITS = {"cr": "crore", "crore": "crore", "crores": "crore",
         "l": "lakh", "lakh": "lakh", "lakhs": "lakh"}
MONEY = re.compile(r"(?:Rs\.?|₹) ?(\d+(?:,\d{2,3})*(?:\.\d+)?)"
                   r"(?: ?((?i:cr|crores?|l|lakhs?))\b)?")
ONES = ("zero one two three four five six seven eight nine ten eleven twelve "
        "thirteen fourteen fifteen sixteen seventeen eighteen nineteen").split()
TENS = "- - twenty thirty forty fifty sixty seventy eighty ninety".split()


def below_100(n):
    if n < 20:
        return ONES[n]
    return TENS[n // 10] + ("" if n % 10 == 0 else "-" + ONES[n % 10])


def indian_words(n):
    """English words with Indian grouping: 120000 -> one lakh twenty thousand."""
    parts = []
    for size, name in ((10**7, "crore"), (10**5, "lakh"), (1000, "thousand"), (100, "hundred")):
        if n >= size:
            parts.append(indian_words(n // size) + " " + name)
            n %= size
    if n or not parts:
        parts.append(below_100(n))
    return " ".join(parts)


def say_number(digits):
    whole, _, frac = digits.replace(",", "").partition(".")
    words = indian_words(int(whole))
    if frac:
        words += " point " + " ".join(ONES[int(d)] for d in frac)
    return words


def say_money(match):
    unit = UNITS.get((match.group(2) or "").lower(), "")
    return " ".join(w for w in (say_number(match.group(1)), unit, "rupees") if w)


def normalize(text):
    for short, full in ALIASES.items():
        text = re.sub(r"\b" + re.escape(short) + r"\b", full, text)
    return MONEY.sub(say_money, text)


for line in ["ASDMA releases Rs 2.5 cr for flood relief",
             "Rs 1,20,000 seized on NH-27",
             "Bridge to cost Rs. 450 crore",
             "₹3 L for each family",
             "Paid Rs 50 Lakh, said police"]:
    print(normalize(line))

Run it with python3 normalizer.py on Python 3.13, and it prints these five lines.

• Assam State Disaster Management Authority releases two point five crore rupees for flood relief
• one lakh twenty thousand rupees seized on National Highway 27
• Bridge to cost four hundred fifty crore rupees
• three lakh rupees for each family
• Paid fifty lakh rupees, said police

What just happened? The regular expression found each amount, stripped the commas and handed the digits to say_number. Its helper indian_words peels off crores, lakhs, thousands and hundreds in turn, so 120000 becomes one lakh twenty thousand.

Three details catch people out. The unit list must include spelled-out and plural words, or "Rs 450 crore" comes out as "four hundred fifty rupees crore".

The amount pattern must not swallow a trailing comma, or "Rs 500, said police" loses its pause. The aliases need word boundaries, so a short form inside a longer word is left alone.

This speller is English only. Each language needs its own, because 2.5 crore is said differently in Assamese, Bengali and Hindi. A misread number is a factual error, so money, death tolls and results get extra tests.

Build each file's key from a hash of its exact spoken text, its voice and an engine version. Any change to the words then produces a new file name, and the old audio is never mistaken for the new.

A hash is a short fingerprint of some data: change one comma and the fingerprint changes completely. ENGINE_VERSION is a label you change whenever you upgrade the model or a voice.

import hashlib
import heapq
import itertools

ENGINE_VERSION = "v7"   # bump when the model or a voice changes


def audio_key(story_id, text, voice_id):
    raw = f"{text}|{voice_id}|{ENGINE_VERSION}".encode("utf-8")
    digest = hashlib.sha256(raw).hexdigest()[:16]
    return f"audio/{story_id}/{voice_id}/{digest}.opus"


stored = set()            # stand-in for the object store
jobs = []                 # stand-in for the job queue (a heap)
order = itertools.count() # tie-breaker: equal priorities keep arrival order


def on_story_saved(story_id, spoken_text, voice_id, breaking=False):
    key = audio_key(story_id, spoken_text, voice_id)
    if key in stored:
        return "reuse " + key
    priority = 0 if breaking else 5
    heapq.heappush(jobs, (priority, next(order), key))
    return "queue " + key


print(on_story_saved(101, "Flood waters rise in the city.", "as-f1"))
print(on_story_saved(102, "Markets open higher today.", "en-m1"))
print(on_story_saved(103, "Cyclone alert for the coast.", "as-f1", breaking=True))
stored.add(audio_key(101, "Flood waters rise in the city.", "as-f1"))
print(on_story_saved(101, "Flood waters rise in the city.", "as-f1"))
print(on_story_saved(101, "Flood waters rise in the city, again.", "as-f1"))
print("render order:", [heapq.heappop(jobs)[2].split("/")[1] for _ in range(len(jobs))])

It prints these lines with Python 3.13. The 16-character fingerprints are the same on any machine, because SHA-256 always gives the same digest for the same input.

• queue audio/101/as-f1/f159e85a5a9d30af.opus
• queue audio/102/en-m1/e731a7e35ecdd47c.opus
• queue audio/103/as-f1/5fb98e3c20a99152.opus
• reuse audio/101/as-f1/f159e85a5a9d30af.opus
• queue audio/101/as-f1/2fcac36b8d2ca0cc.opus
• render order: ['103', '101', '102', '101']

What just happened? The second save of story 101 found its key in the store and reused the file. Adding ", again" changed the hash, so the corrected text got a new key. The breaking story 103 jumped to the front.

One gap to close in a real worker: two quick saves of the same text queue the same key twice. The worker should check the store again and skip a key that already exists.

Do not bump ENGINE_VERSION for a lexicon fix, or every story in the catalogue re-renders. Hash the text after pronunciation lookup instead, so only stories containing the fixed name get new keys.

Also apply Unicode NFC normalization before hashing. The same Assamese or Bengali vowel sign can arrive as one code point or two, and each form gives a different key.

Why the queue needs a counter

The heap sorts tuples item by item. Real jobs are usually dicts holding the text and voice. If two jobs share a priority, Python compares the dicts and raises TypeError, because dicts do not support less-than. We ran this with queue.PriorityQueue.

The fix comes straight from Python's heapq documentation: put a unique, increasing count between the priority and the job. Ties then keep arrival order, and Python never compares two jobs.

Why new names beat clearing the CDN

CloudFront's documentation is a clear example. It keeps serving cached copies until they expire or you invalidate them, which means asking its servers to drop them. It recommends versioned file names instead, so you neither wait nor pay for invalidations.

This only works if the app asks for the story's current key each time. Keep the story's metadata on a short cache time, and keep the audio files themselves cacheable for a long time.

The story record should point only at a finished file. A text correction clears that pointer until the new file is uploaded, so the app shows the audio as updating, or plays fast-lane chunks for breaking news. It never falls back to old audio.

When a render finishes, the worker sets the pointer only if its key still matches the story's current text. A late render of an older version is kept but never pointed at.

An engine upgrade is different, because the old audio is still correct. Keep the old key live until the re-render lands, then switch the pointer.

Split the story into sentences, render the first ones immediately, and let playback begin while the rest is still being made. A separate high-priority lane keeps these jobs from waiting behind normal stories.

Two measures matter here. Time to first audio is how long a listener waits after tapping play before hearing anything. Real-time factor is render time divided by audio length, so 0.1 means ten seconds of speech per second of compute.

Here is the cyclone alert after the redesign, from the editor's click to a listener's phone. The timings are made up, but the order of events is the design.

1. 1

   The editor publishes the alert

   The 300-word story is marked as breaking. The publishing tool sends a job to the queue with the top priority.

2. 2

   The job skips the line

   Two GPU workers are reserved for the fast lane only. One picks up the job within a second, even with 200 normal jobs waiting.

3. 3

   The first sentence goes out

   The worker normalizes the text, renders the headline and first sentence, and uploads it as chunk 1. The first listeners hear audio about 1.5 seconds after the alert is published.

4. 4

   The rest streams behind it

   The app then asks for chunk 2, chunk 3 and so on. With a real-time factor well below 1, each chunk is ready before the previous one ends.

5. 5

   The full file is saved

   When the last sentence is done, the worker encodes the whole story as one Opus file under its normal key. From then on, every play comes from the CDN.

Split at sentence ends, not in the middle. The model only sees the chunk you send and treats it as a whole sentence, so a cut in the middle can end on the wrong tone. A short pause between chunks makes the joins sound natural.

You measure voice quality mostly by asking people to listen and rate it, then add automatic checks for pronunciation, speed and failures. No single number tells you a voice is good enough for news.

The common listening test gives a mean opinion score, or MOS. Listeners rate clips from 1 to 5 and you average the ratings. In the 2017 paper, each clip got at least 8 ratings in half-point steps.

Use native speakers of each language as raters. Someone who does not speak Assamese cannot judge whether an Assamese sentence sounds natural. A small panel of regular listeners per language, tested monthly, is a good start.

• A test set of hard sentences per language, full of names, numbers, dates and mixed-language quotes.
• A pronunciation check: raters mark each name in the test set right or wrong, which gives an error rate.
• Time to first audio and render time per story, tracked on the monitoring dashboard.
• Failed renders and silent files, caught by comparing each file's length with its word count.

The length check is cheap and catches real failures. At 120 to 180 words a minute, a 350-word story should last about 2 to 3 minutes. A 10-second file means something broke, so the worker retries before uploading.

You can check a file yourself with ffprobe -v error -show_entries format=duration,size,bit_rate story.opus. We ran it with ffmpeg 6.1.1 on 19 October 2026. A 149-second speech file at 32 kbit/s came to 613 kB, with a bit_rate of about 32,900 bits per second.

Run the full test set before any change goes live, even a lexicon update. A change that fixes one name can quietly break another.

The main risks are reading facts wrongly, misleading listeners about who is speaking, and copying a real person's voice without consent. Each needs a clear rule, decided before launch.

A misread number changes the story even when the text was right. Reading "Rs 2.5 cr" as "2.5 lakh" is a factual error. That is why amounts sit in the hard-sentence test set.

Listeners should know the voice is synthetic. A short line before each story saying it is read by a computer voice is enough. Never let listeners think a real anchor recorded it.

Voices themselves need consent. Train each voice only on recordings from a paid voice artist who signed a written agreement. It should cover how the voice may be used and how the artist can withdraw it.

Explain it as a speech pipeline plus a delivery system, and let the type of text drive every choice. Walk through these steps in order.

1. 1

   Ask about the text

   Is it fixed content heard by many people, or new text for each listener? How many languages, and how fresh must the audio be?

2. 2

   Draw the speech pipeline

   Text normalization, pronunciation lookup, acoustic model, vocoder, compressed audio. Say that many errors start in the front end, not the models.

3. 3

   Choose when to render

   Render on save for fixed content, and stream for live text. Back it with rough maths: renders per day against plays per day.

4. 4

   Design storage and delivery

   Object storage plus a CDN, with keys built from a hash of the text, voice and engine version. Explain how this makes corrections safe.

5. 5

   Add a fast lane

   A priority queue with a tie-breaking counter and reserved workers for urgent text. Render sentence by sentence and track time to first audio.

6. 6

   Close with quality and safety

   Native-speaker listening tests, a hard-sentence test set, length checks, voice consent and clear labels that the voice is synthetic.

A common follow-up is what happens when the model is upgraded. Bumping the engine version gives every story a new key. Old files keep playing until each re-render lands, newest stories first.