Speech Recognition System Design: Hinglish Call Transcripts

People can only listen to a tiny share of calls, so the call that breaks a rule is usually the one nobody heard. Speech recognition turns every call into text that a small team can search. Speech recognition system design is how you build that at scale.

Picture a common setup. An insurance company's outsourced call centre, a BPO, in Noida takes about 40,000 calls a day, averaging 6 minutes each. The setup is made up, but the problem is real.

Twelve auditors can listen to about 800 calls a day. That is 2 percent of the floor. When a customer complains that an agent promised a guaranteed return on a policy, nobody has heard that call.

The quality team wants one thing. Type a phrase like guaranteed return and see every call where an agent said it, with the second it was said.

The naive first try sends every recording to an English-only speech model and indexes whatever comes back. It looks simple, and it fails in three ways.

• Mixed language. A customer says premium kitna hai, meaning how much is the premium. An English-only model may write sound-alike English words that mean nothing.
• Leaks. Customers read out card and policy numbers, so plain text search shows them to anyone with access.
• Misleading scores. The first accuracy number looks terrible, yet many errors are only spelling variants, like nai written for nahi.

The rest of this guide rebuilds the system one decision at a time. Each fix answers one of these failures.

You need six words. Each comes with an example from the call centre.

• ASR, automatic speech recognition: software that turns spoken audio into written text. Speech-to-text means the same thing.
• WER, word error rate: the share of words the system got wrong against a careful human transcript. A WER of 0.20 means about 1 word in 5 is wrong.
• Code-switching: moving between languages inside one conversation, often inside one sentence. Example: Aapka claim approve ho gaya hai, documents email kar diye.
• Diarization: working out who spoke when in a recording. Example: from 00:42 to 00:51 the customer speaks, then the agent replies.
• VAD, voice activity detection: finding the parts of the audio that hold speech, so long silences never reach the model.
• Confidence score: a number, usually from 0 to 1, that the model attaches to a word or segment to show how sure it is.

Speech recognition system design is planning every part that turns raw audio into useful text at scale. That means storage, a job queue, audio cleanup, the model, post-processing, search and review.

Interviewers ask it to see whether you think past the model. A great model inside a weak pipeline still loses calls, leaks card numbers or returns text nobody can search.

Start with requirements written as numbers. Transcripts must be ready within 15 minutes of a call ending. The quality team must search all calls from the last 90 days. No card or Aadhaar number may appear in any transcript.

Then a rough load estimate. 40,000 calls of 6 minutes is 240,000 minutes, or 4,000 hours of audio a day. The team designs for four times that, 16,000 hours, to leave room for growth.

Here is the system. Follow one call from the recorder on the left to the dashboard at the end.

The job queue keeps the system calm. Calls finish in bursts, often at shift change, and workers pull jobs at their own pace. If a worker crashes before it confirms a job, the queue hands that job to another worker.

Each stage is a separate pool because each needs different hardware. Audio prep and post-processing run on cheap CPU machines. Only the ASR step needs GPUs.

Mono recordings, where both voices share one channel, need a diarization step instead. A 2021 review of speaker diarization sums up the task as identifying who spoke when.

An ASR model turns audio into words in three moves. It converts sound into rows of numbers, scores possible text pieces for every short slice of time, and then searches for the most likely sentence.

Think of the typist who sits outside a tehsil court, typing applications as people dictate them. His ear is the feature step. His guess at each sound is the encoder. His sense of how a court form reads is the language model.

Where the picture breaks: the typist understands meaning. The model only learns which text tends to follow which sounds, so it can write a fluent sentence that is confidently wrong.

Step 1: from sound to features

Classic phone lines carry audio at 8,000 samples per second, the nominal rate in the ITU-T G.711 standard. Prep code cuts the audio into short overlapping windows, often 25 milliseconds long, one every 10 milliseconds.

For each window it measures the energy in each pitch band, on a scale close to how ears hear. These rows are called log-mel filterbank features. With 25 ms windows every 10 ms and no padding at the edges, one second of audio gives 98 rows.

Step 2: the acoustic encoder

The encoder is a neural network, a model made of many layers of simple maths learned from examples. For each slice of time it outputs a score for every possible text piece, such as a letter or a common chunk of a word.

One well-known encoder design comes from a 2020 paper. It mixes attention, which links sounds far apart in time, with convolution, which picks up short local patterns.

On LibriSpeech, about 1,000 hours of read English audiobooks, that paper's best model reported 2.1 percent WER on the clean test set without a language model. Keep that number in mind: it is clean read English, not a noisy phone line.

Step 3: aligning and decoding

The audio has about 100 slices a second, but people say only a few words a second, and nobody labels which slice belongs to which letter. One common method, CTC (connectionist temporal classification), avoids choosing one line-up. During training it adds up the scores of every line-up that spells the right text.

CTC adds a special blank token that stands for nothing. The model scores a letter or a blank for every slice. Then repeats are merged and blanks removed, so h h blank e l blank l o collapses to hello.

Pause and predict: what does h h e l l o collapse to, with no blank between the two l's? It gives helo, because the repeated l merges. That is why CTC needs the blank.

A decoder then runs beam search: it keeps the few most likely sentences so far and extends them slice by slice. A separate language model scores how natural each sentence sounds. That is how premium kitna hai beats a sound-alike English phrase.

CTC is one of several ways to align audio with text. Here is the path inside one ASR worker. Look at the box feeding the decoder from the side, where the team adds its own words.

The typist knows words like khatauni, affidavit and vakalatnama, so he rarely mishears them. The team's version is a custom vocabulary of plan names and rider names, given extra weight during decoding.

Hinglish is hard because the model must handle two languages in one sentence, and because Hindi words in Roman letters have no fixed spelling. Most public training audio is also cleaner than a call-centre line.

Researchers treat this as its own problem. A 2021 challenge for Indian languages included a Hindi-English code-switching task. Its six baseline set-ups scored WERs from 24.7 to 44.3 percent on the Hindi-English test and blind sets.

That data came from recorded technical tutorials at 16,000 samples per second. Noisy 8,000-sample phone calls are a different setting, so do not borrow that number. Measure your own.

The same paper added a second score, transliterated WER. English words in its transcripts appeared in both Roman and native scripts, and plain WER counts the same word in another script as an error.

The team lists the specific problems in a week of calls.

• Mixed grammar. Hindi word order with English nouns, like policy ka status kya hai. A model trained on one language expects the wrong next word.
• Many spellings. The same word comes out as nahi, nahin or nai. Each looks like an error when you score, and each breaks search.
• Script choice. Hindi words could be written in Devanagari or Roman letters. Auditors search by typing Roman letters on English keyboards.
• Numbers. Customers say do sau pachaas or two fifty or 2-5-0 for the same amount. Policy numbers get read out one digit at a time.
• Phone audio. Narrow 8 kHz lines, traffic noise, a TV in the room and two people talking at once.

The fixes are boring, and they work. Pick one script, Roman letters, because that is how people will type searches. Build a spelling list that maps common variants to one form.

Then add inverse text normalisation, a step that turns spoken numbers into digits, so do sau pachaas becomes 250. Masking and search both depend on it.

The biggest gain comes from your own data. The auditors correct 300 hours of real calls, and the team fine-tunes, or trains further, a multilingual model on them. WER on these calls drops from 0.31 to 0.17 (made up).

Transcribe in batch when people read transcripts after the call, and stream only when someone needs the words during it. Batch is usually cheaper and more accurate, because the model hears the whole segment before deciding.

Streaming ASR sends partial results, words that may still change, then a final result when a phrase ends. Its encoder sees little or no future audio, and that costs accuracy.

One 2020 study measured the gap on the same encoder design. On the LibriSpeech clean test set, WER rose from 2.3 percent with the whole utterance visible to 4.6 percent in streaming mode. The paper's own method narrowed that gap, to 2.5 against 3.7 percent, but did not close it. Part of that came from WER with the whole utterance visible slipping from 2.3 to 2.5 percent.

The typist shows the difference. Let him hear the whole application first and he types it right once. Make him type while you speak, and he types a word, hears the rest, and goes back to fix it.

The quality team needs search and audits, so batch comes first. A live agent-assist feature, such as reminding agents to read a disclosure, can come later as a separate streaming path into the same transcript store.

You get the GPU count from hours of audio per day and the real-time factor (RTF), processing time divided by audio length. An RTF of 0.01 means one hour of audio takes 36 seconds.

Measure RTF yourself, because the GPU, the batch size and the model all change it. Time a fixed 10-hour slice of your test set, then divide the seconds of compute by 36,000 seconds of audio.

Say the team measures an RTF of 0.01 (made up). For 16,000 hours a day, that is 160 GPU-hours of work.

Calls are not spread evenly. Most land between 9 am and 9 pm, so 160 GPU-hours over 12 hours means 13.3 GPUs busy for those 12 hours. Round up to 14, and plan for bursts at twice that: 28 GPUs at peak.

At night the queue empties. The autoscaler, a controller that adds or removes machines with load, shuts most GPUs down.

• Batch segments of similar length. A batch of 2, 3, 28 and 30 second segments pads every one to 30 seconds, wasting 57 seconds of GPU work. Two batches of similar lengths waste 3.
• Measure how much of each call is silence, and cut it with VAD before the GPU step. Every minute cut is a minute you do not pay for.
• Set the queue's visibility timeout, how long a taken job stays hidden from other workers, above your slowest job. Amazon SQS, a managed queue service, defaults it to 30 seconds.
• Key every write by call id, so a job that runs twice overwrites one transcript instead of adding a copy. SQS standard queues can deliver a message more than once.
• Track queue age, the time the oldest job has waited. Alert at 10 minutes, before the 15-minute promise breaks.

Check what the recorder really sends before you size anything. This ffprobe command, part of the free FFmpeg tools, prints the codec, sample rate and channel count of one recording.

ffprobe -v error -show_entries stream=codec_name,sample_rate,channels -of default=nw=1 call.wav

On a stereo 8 kHz WAV file of 16-bit samples you should see three lines: codec_name=pcm_s16le, sample_rate=8000 and channels=2 (checked on FFmpeg 6.1.1). A G.711 file shows pcm_mulaw or pcm_alaw instead.

You measure accuracy with word error rate. Line up the system's text against a careful human transcript, count the wrong, missing and extra words, and divide by the number of words in the human transcript.

Those three errors have names. A substitution is a wrong word, a deletion is a missed word, and an insertion is an extra word, often from noise. Because insertions count, WER can go above 1.

NIST, the US standards body, publishes the SCTK toolkit with a scorer called sclite. Its documentation says it aligns the two texts with dynamic programming, the table-filling method below. It costs insertions and deletions 3 and substitutions 4.

Here is a small WER function in plain Python. It finds the fewest edits that turn the human words into the system's words. This is the classic edit distance method, used on words instead of letters.

def wer(reference, hypothesis):
    ref, hyp = reference.split(), hypothesis.split()
    # d[i][j] = fewest edits that turn the first i ref words into the first j hyp words
    d = [[0] * (len(hyp) + 1) for _ in range(len(ref) + 1)]
    for i in range(len(ref) + 1):
        d[i][0] = i                      # i deletions
    for j in range(len(hyp) + 1):
        d[0][j] = j                      # j insertions
    for i in range(1, len(ref) + 1):
        for j in range(1, len(hyp) + 1):
            cost = 0 if ref[i - 1] == hyp[j - 1] else 1
            d[i][j] = min(d[i - 1][j] + 1,          # deletion
                          d[i][j - 1] + 1,          # insertion
                          d[i - 1][j - 1] + cost)   # substitution or match
    return d[len(ref)][len(hyp)] / max(len(ref), 1)

Each cell d[i][j] stores the fewest edits so far. The min line tries a deletion, an insertion and a substitution and keeps the cheapest. The last line divides by the reference length, and max avoids dividing by zero.

Now the Hinglish catch. The team's first scores looked terrible, so they checked by hand. Many errors were not errors at all: the model wrote nai where the auditor typed nahi. So normalise both texts before scoring.

import re

SPELLINGS = {"nahi": "nahin", "nai": "nahin", "pls": "please",
             "plz": "please", "ok": "okay", "acha": "accha"}

def normalise(text):
    words = re.findall(r"\w+", text.lower())   # lower case, drop punctuation
    return " ".join(SPELLINGS.get(w, w) for w in words)

ref = "Policy renew nahi hua, please check karo"
hyp = "policy renew nai hua plz check karo"
print(round(wer(ref, hyp), 2))                        # raw text
print(wer(normalise(ref), normalise(hyp)))            # after normalising

Put both blocks in one file and run it. You should see 0.57, then 0.0. What just happened? The raw score counted a capital letter, a trailing comma and two spelling variants as four wrong words out of seven.

Use the same normalise function for scoring and for search, so both see one spelling. It is written for Roman-script text. Python's word pattern splits Devanagari words at their vowel signs.

Two standard scorers give two answers

Check your scorer against a library before you trust it. jiwer is a Python WER library. With version 4.0.0, this prints 0.5714285714285714, the same as the function above.

pip install jiwer==4.0.0
python -c "import jiwer; print(jiwer.wer('Policy renew nahi hua, please check karo', 'policy renew nai hua plz check karo'))"

sclite from SCTK 2.4.10 scores the same raw pair at 3 errors, not 4, because it ignores case unless you pass its -s flag. With -s it agrees: 4 substitutions out of 7 words.

Its 3-and-4 costs can also change the count itself. We ran sclite -s on 20,000 random short word pairs. It counted more errors than the function above, where every edit costs 1, on 32 pairs and never fewer.

So write down which tool, version and flags produced every WER you report. Never compare numbers from two scorers.

WER alone does not tell the quality team everything. A missed haan barely matters, but a missed nahin flips the meaning. Also track keyword recall: of every time an agent truly said guaranteed return, how often does search find it?

You protect customer data by masking sensitive numbers in both the text and the audio, limiting who can read transcripts, and deleting data once its purpose is served. Search makes a leak easier, so the rules must be strict.

Card rules are the sharpest. The card verification code is the short security code printed on a card. A PCI Security Standards Council FAQ says keeping it in any digital audio recording after authorisation breaks PCI DSS, the card industry's security standard.

The same FAQ says that where technology exists to suppress or redact audio during card entry, it should be enabled. So masking the text alone is not enough; the audio needs muting too (as of 19 October 2026).

India's Digital Personal Data Protection Act, 2023 requires reasonable security safeguards to prevent a personal data breach. Section 8(7) requires erasing personal data once its purpose is no longer served, or when the person withdraws consent, unless a law requires keeping it.

The government notified the Digital Personal Data Protection (DPDP) Rules, 2025 on 14 November 2025, with an eighteen-month period for phased compliance. Check which duties apply on the day you build (as of 19 October 2026).

Every segment also passes a confidence check. Look at the two question boxes below: one decides whether a person must fix the text, the other whether digits must be masked.

The masking step uses two pattern rules. Run them on normalised text, where spoken digits are already numerals.

import re

# Rule 1: 6 or more digits in a row, allowing one space or dash between them
LONG_RUN = re.compile(r"\d(?:[ -]?\d){5,}")
# Rule 2: a 3 to 6 digit code right after a trigger word such as cvv, otp or pin
SHORT_CODE = re.compile(r"\b(cvv|otp|pin)\b(\D{0,12}?)\d(?:[ -]?\d){2,5}\b")

def mask_digits(text):
    text = LONG_RUN.sub("[NUMBER]", text)
    return SHORT_CODE.sub(r"\1\2[CODE]", text)

print(mask_digits("card number 4 5 3 2 1 1 2 2 3 3 4 4 5 5 6 6 hai"))
print(mask_digits("policy number 5012-3345-88 renew karna hai"))
print(mask_digits("cvv hai 4 1 7"))
print(mask_digits("premium 250 rupees, okay 4 1 7"))

Run it and you should see four lines. The card and policy numbers become [NUMBER], the cvv becomes [CODE], and the last line comes back unchanged. That last line is the gap.

A customer who reads a three-digit code with no trigger word slips past both rules. So the design also mutes the audio of every masked word, using its timestamps. Where the dialler, the phone system that places and records calls, allows it, card details are taken off the recorded line.

The rules also mask a bare six-digit amount like 120000 and dates read as digits. That false alarm is the price of catching long numbers. Amounts written with commas, like 1,20,000 (1.2 lakh, that is 120,000), survive.

A small tagging model catches names and addresses that patterns miss. Access is narrow too. An auditor sees only her own client's calls, raw audio is deleted after 90 days, and every search is logged with who ran it.

You make transcripts searchable by storing them as short segments with call id, speaker, start time and confidence, then indexing the normalised, masked text. A search hit then opens the recording at the exact second.

An auditor types guaranteed return and gets 37 calls from last week (made up). She clicks one, and the player jumps to 03:12, where the agent says it. She listens to ten seconds instead of six minutes.

Keyword search on normalised text handles most queries. Some auditors search by meaning, such as the agent promised a fixed return. For those, combine keyword search with search by meaning, a mix called hybrid search.

Two details matter. Index only masked text, so search does not surface the numbers the rules catch. Store the speaker on each segment, so auditors can search only what agents said.

Explain it as a pipeline around a model, and justify each part with a number or a failure it prevents. Here is an order that works.

1. 1

   Clarify the use

   Ask who reads the text, how soon, in which languages, and how much audio arrives per day. This picks batch or streaming.

2. 2

   Estimate the load

   Turn calls per day into hours of audio, then into GPU-hours using a measured real-time factor. Add a peak factor for shift-change bursts.

3. 3

   Draw the pipeline

   Object storage, a job queue, audio prep with VAD and diarization, ASR workers, post-processing, a transcript store and a search index.

4. 4

   Open the model box

   Features, encoder, CTC or another aligner, beam search, and a language model with a custom vocabulary.

5. 5

   Handle the language

   Explain code-switching, one script, a spelling list and inverse text normalisation for numbers.

6. 6

   Measure and protect

   WER on your own normalised test set with a named scorer, keyword recall, masking in text and audio, access limits and deletion rules.