15 minute read

TL;DR: Standard ASR benchmarks test clean, read speech in studio conditions. Voice agents operate on noisy phone channels, disfluency-laden conversation, and domain-specific vocabulary, none of which standard benchmarks like LibriSpeech measure. The Back to Basics paper (arXiv:2603.25727) quantifies the gap across seven production ASR systems: robustness does not transfer across languages, environments, or acoustic conditions. Understanding the five structural gaps between benchmark and production is the first step to building voice agents that actually work.

A lavalier microphone in a noisy café environment, illustrating the gap between controlled benchmark conditions and real-world voice agent deployments


The Whisper paper reports 2.7% Word Error Rate on LibriSpeech test-clean for Whisper large-v2 (large-v3 shipped without a paper). NVIDIA’s Parakeet-TDT-0.6B-v2 model card reports 1.69%. These numbers circulate in procurement decisions, integration announcements, and architecture reviews. They feel like permission to stop worrying about speech recognition.

Then your voice agent ships. Users call from cars, offices, noisy restaurants. They say “um” and restart sentences. They use your product’s actual name, which the model has never seen. And error rates on the hardest calls land far above the benchmark. On the WildASR benchmark’s real-speech conditions, English WER on short utterances ranged from 38.7% to 73.9% across seven systems. Not because the models are bad. Because the benchmark was measuring something else entirely.

What LibriSpeech actually tests

LibriSpeech is a 1,000-hour corpus of audiobook recordings, narrated by volunteers in reasonably quiet conditions, in standard American and British English. The speakers are reading. There are no disfluencies, no background noise, no phone channel degradation, no accents outside the corpus’s demographic range.

It is an excellent benchmark for what it measures: how accurately a model transcribes prepared, articulate speech in a controlled acoustic environment. That is a real use case. It is not a voice agent.

The gap isn’t a criticism of LibriSpeech. It was designed for controlled comparison between models, and it works well for that purpose. The problem is how the scores travel. A vendor quotes 2% or 3% WER and the number lands in an engineering decision with none of its context. “Near-human accuracy” becomes the implicit conclusion. The benchmark’s scope evaporates.

Here is what production conditions actually look like:

Condition Benchmark assumption Production reality
Audio quality 16 kHz close microphone 8 kHz phone channel, Bluetooth, varying mics
Background noise Near-silent Streets, offices, cars, hospitals
Speech style Read, fluent Conversational, disfluent, very short turns
Vocabulary General English Product names, domain jargon
Processing mode Batch, full audio context Streaming, a few hundred ms of lookahead
Speaker range Adult readers, limited accents Children, older callers, global accents, code-switching

How much each condition costs depends on the model, and the costs do not add up neatly. In a voice agent call, several apply at once. The WildASR analysis has measured numbers for several of these conditions on real speech.

Five structural gaps, and what the Back to Basics paper measured

The March 2026 paper “Back to Basics: Revisiting ASR in the Age of Voice Agents” (arXiv:2603.25727, Boson AI) evaluates seven widely used ASR systems using WildASR, a diagnostic benchmark built from real human speech in English, Chinese, Japanese and Korean. WildASR tests robustness along three axes: environmental degradation, demographic shift, and linguistic diversity.

The findings are worth sitting with. Model robustness does not transfer across languages or conditions. A model that handles degraded English well can fall apart in Chinese, Japanese or Korean under the same perturbation, and rankings change from one condition to the next. There is no safe assumption that a model’s English benchmark score predicts its behavior in another language, another accent, or another noise environment.

The paper organizes its tests by those three axes. The five gaps below are this post’s framing of what voice agents need and benchmarks miss, with the paper’s evidence where it applies:

Gap 1: Disfluency handling. Spontaneous speech is full of fillers, repetitions, false starts and revisions; classic studies of conversational English count roughly six disfluencies per 100 words. LibriSpeech, being read audiobooks, has very few. Models trained predominantly on read speech handle disfluencies by deleting them, converting them to repeated words, or substituting semantically plausible alternatives. “I want to um… schedule my return” becomes “I want to schedule my return” on a good day, and “I want to order my return” on a bad one. The direct WER impact from disfluencies alone is modest, but the downstream intent corruption is significant.

Gap 2: Channel degradation. A customer calling from a car on a phone network sends audio sampled at 8 kHz rather than the 16 kHz standard for modern ASR. One study of Hindi telephony speech found IndicWav2Vec degrading from 11.60% WER on clean read Hindi to 41.72% on phone audio from Gram Vaani, which the authors use as the closest public proxy for clinical telephony (arXiv:2512.16401). A contact center voice agent handling 10,000 calls per day will see this condition on a meaningful fraction of them.

Gap 3: Domain vocabulary. ASR training corpora skew heavily toward general English text. Your product name, your plan tiers, your competitors’ names, your industry’s jargon: all out-of-vocabulary. Slot Error Rate (accuracy on the specific values that downstream tasks depend on) is usually higher than WER; Deepgram’s January 2026 SER guide puts the gap at roughly 6 to 12. For a voice agent processing return requests, the flight number matters more than the filler words.

Gap 4: The streaming tax. Voice agents cannot wait for users to finish sentences before transcribing. In conversation between people, the typical gap between turns is around 200ms (Stivers et al., 2009, PNAS), so streaming ASR has to commit to words with only a few hundred milliseconds of lookahead. A batch model that reads the whole utterance has more context and is usually more accurate. The size of that gap depends on the model and the latency setting, so measure it at the setting you will ship.

Gap 5: Hallucination under degraded audio. This is the least discussed and most dangerous gap. When audio is partially degraded or contains near-silence, models do not return low-confidence outputs. They generate plausible transcriptions that were never spoken. “Cancel my order” can become “Schedule my order” under acoustic pressure. Both are grammatical. Only one matches user intent. The SHALLOW benchmark (arXiv:2510.16567) categorizes these hallucinations across lexical, phonetic, morphological, and semantic dimensions. WildASR found models hallucinating plausible but unspoken content under partial or degraded input, and introduced a hallucination error rate to measure it. In one example, “no I can” became “no I can’t”. The failure varies by language, condition and speaker group in ways English benchmark scores do not predict.

flowchart LR
    A["LibriSpeech benchmark\n(~2-3% WER)"] --> B["Whisper Large-V3\nin lab"]
    
    C["Production call\n(phone channel)"] --> D["8 kHz audio\n+channel noise"]
    D --> E["Domain vocabulary\ngap"]
    E --> F["Streaming latency\nconstraint"]
    F --> G["Voice agent\nmuch higher WER"]
    
    B -.->|"several conditions\nat once"| G
    
    style A fill:#e8f5e9
    style G fill:#ffebee
    style B fill:#e8f5e9

Why WER is the wrong primary metric for voice agents

Word Error Rate measures the edit distance between the ASR transcript and a reference transcription, normalized by reference length. It treats every word as equally important. For voice agents, this is wrong in two directions.

First, filler words matter less than content words. Getting “um” wrong is not the same as getting “account number” wrong. A WER of 8% could mean the model is flawlessly capturing all named entities and occasionally mangling hesitations, or it could mean it’s misreading critical values reliably. WER cannot tell the difference.

Second, Slot Error Rate is the metric that actually predicts downstream task success. SER measures whether the model correctly transcribed the tokens that matter: names, numbers, dates, addresses, product identifiers. A contact center voice agent cares whether it captured “return order 7721-B,” not whether it handled the surrounding conversation fluently. Measure it on your own slots; Deepgram’s January 2026 guide puts SER roughly 6 to 12 above WER on domain vocabulary.

The latency budget compounds this. Human turn gaps cluster around 200ms, and pauses much longer than that start to feel like hesitation; past a second, the agent feels broken. A voice agent optimizing for low latency (as all production voice agents must) is permanently operating in the streaming accuracy regime, not the batch regime.

A useful rule of thumb: if your total voice agent round-trip budget is 800ms and LLM inference takes 400ms, your ASR layer has roughly 150ms. At that latency, you are not running Whisper Large-V3 in batch mode. You are running a streaming endpoint at a fraction of the model’s maximum capability.

What WildASR reveals about model selection

The WildASR benchmark’s most actionable finding is the robustness transfer problem. Teams often evaluate a single ASR model on English benchmarks and assume multilingual performance scales predictably. It does not.

WildASR tests models independently along three axes. A model robust to environmental degradation in English may fail badly under the same conditions in Chinese, Japanese or Korean. Qwen2-Audio, for example, had the lowest error of the seven systems on accented Chinese and the highest on Korean code-switching. Robustness on one axis does not predict robustness on another, so each needs its own evaluation.

For teams building multilingual voice agents, or voice agents serving diverse user populations in any single language, this has direct practical implications:

  • Evaluate each language independently on realistic samples from your actual user base
  • Test each degradation axis (noise, channel, accent, disfluency) separately before combining
  • Do not assume English robustness predicts other-language robustness
  • Use WER only as a secondary signal; measure SER and task completion as primary signals

The paper’s argument is that without diagnostics that isolate one failure factor at a time, practitioners cannot predict which conditions, in which languages, will cause how much degradation.

How to actually evaluate ASR for your voice agent

Standard ASR evaluation for voice agents requires a different protocol than benchmark comparison. Here is a practical starting checklist:

ASR production readiness checklist

Channel:
  [ ] Test on your actual delivery channel (phone at 8kHz, WebRTC, mobile app mic)
  [ ] Test at 10 dB SNR and 5 dB SNR for worst-case calls
  [ ] Include 2% packet loss simulation for VoIP deployments

Vocabulary:
  [ ] Build a domain-specific test set with your product names, plan names, jargon
  [ ] Measure Slot Error Rate, not just WER, on named entity tokens
  [ ] Test your word-boosting / custom vocabulary implementation

Speech style:
  [ ] Include at least 20% disfluent samples (fillers, false starts, repetitions)
  [ ] Include non-native speaker samples at your expected accent distribution
  [ ] Test at your streaming latency target (not batch)

Latency:
  [ ] Measure WER at your actual latency constraint (e.g., 150ms for an 800ms budget)
  [ ] Confirm streaming endpoint, not batch, is what gets evaluated
  [ ] Test under concurrent load conditions (WER often degrades under load)

Failure mode coverage:
  [ ] Test silent frames and near-silence for hallucination
  [ ] Test domain-critical pairs: cancel/schedule, return/retain, yes/no
  [ ] Monitor SER continuously in production, not just WER

The acoustic conditions your users call from are the benchmark that matters. LibriSpeech tells you the ceiling. Your production test set tells you where you’ll actually land.

For teams using streaming ASR architectures, the latency-accuracy tradeoff is built into the pipeline design. There is no avoiding it, only making it explicit. For the full picture of where ASR failure compounds with turn-taking and agent reasoning failures, the τ-Voice benchmark analysis documents how these errors cascade through a complete voice agent conversation. The τ-Voice results (best voice agent at 38% task completion under realistic conditions, upper bound of a 26-38% range across tested systems) are partly an ASR story. The transcription failures at authentication cascade into every subsequent tool call.

Key takeaways

  • A 2-3% WER on LibriSpeech does not predict your production accuracy. On WildASR’s real-speech conditions, error rates of 20% to 70% are common for widely used systems.
  • The five structural gaps are: disfluency handling, channel degradation, domain vocabulary, the streaming latency tax, and hallucination under degraded audio. Each adds independently to production WER.
  • Slot Error Rate, not WER, is the right primary metric for voice agents. It is usually higher than WER on domain vocabulary.
  • The WildASR paper’s key finding: model robustness does not transfer across languages or conditions. Evaluate each independently.
  • Streaming ASR usually costs some accuracy compared to batch. At a 150ms ASR budget inside an 800ms round-trip, you are not running in the benchmark regime, so measure at your real latency setting.
  • Build your own production test set. Include phone-channel audio, disfluent samples, domain vocabulary, and your actual accent distribution. Then run it at your latency constraint.

FAQ

Why does ASR perform so much worse in production than benchmarks suggest? ASR benchmarks like LibriSpeech test clean, read speech. Production voice agents encounter phone-channel audio (8 kHz vs 16 kHz), background noise, natural disfluencies (um, uh, false starts), short turns and domain-specific vocabulary. Several of these apply at once on a real call. The WildASR benchmark measured some of them on real speech: across seven systems, English WER on short utterances ranged from 38.7% to 73.9%, against an average near 6% on clean FLEURS audio.

What is the streaming accuracy tax in ASR? Streaming ASR has to commit to words within a few hundred milliseconds, before the user finishes speaking, so the model sees less future context than a batch model that reads the whole utterance. That usually costs some accuracy, and the size of the gap depends on the model and the latency setting. Voice agents need streaming, so measure WER at the latency setting you will actually run, not in batch mode.

What is WildASR and how does it differ from LibriSpeech? WildASR (arXiv:2603.25727, Boson AI, 2026) is a diagnostic benchmark in English, Chinese, Japanese and Korean, sourced from real human speech, that independently measures robustness along three axes: environmental degradation, demographic shift, and linguistic diversity. LibriSpeech uses read audiobooks in clean studio conditions. WildASR found that model robustness does not transfer across languages or conditions. A model robust in English degrades unpredictably in other languages under the same acoustic stress.

What is Slot Error Rate and why does it matter more than WER for voice agents? Slot Error Rate (SER) measures transcription accuracy on the specific values that matter to downstream tasks: product names, account numbers, dates, addresses. It is usually higher than WER, because one wrong word makes a multi-word slot wrong and because domain vocabulary is rare in ASR training data; Deepgram’s developer guide puts the gap at roughly 6 to 12. A voice agent booking a flight can tolerate filler words being transcribed imperfectly; it cannot tolerate the flight number being wrong.

How should teams evaluate ASR for voice agent production readiness? Use domain-specific test sets with real recordings from your target channel (phone, web, mobile). Measure Slot Error Rate alongside WER, test at your target streaming latency, include disfluency-heavy samples and non-native accents, and simulate your noisiest expected conditions. Benchmark WER on LibriSpeech predicts almost nothing about whether your voice agent will work.


Further reading