11 minute read

You build a voice agent, test it with your own voice in a quiet room, and it sounds great. Then it hits users and you discover the agent loses track of domain policies, stumbles on interruptions, and fails transfers. τ-Voice is the benchmark that surfaces these failures before they reach users.

TL;DR: τ-Voice tests voice agents on 278 customer-service tasks (retail, airline, telecom) under realistic audio conditions. The best voice agent completes 38% of tasks, against 85% for a reasoning text model (GPT-5) and 54% for a non-reasoning one (GPT-4.1). Failures concentrate on transcription errors during authentication, logical errors, claimed actions that never happened, and accent handling. This post explains what the benchmark measures and gives you a checklist to pressure-test your own agent before shipping.

A laboratory testing station with two precision measurement microphones facing each other across a calibrated gap, analog VU meters and spectrum an...


What existing voice agent benchmarks miss

Standard ASR and TTS benchmarks measure one thing: the quality of a single transformation. Word Error Rate tells you how accurately speech becomes text. MOS scores tell you how natural synthesized speech sounds. Neither tells you whether your agent can book a flight.

Voice agent failures don’t happen at the transcription layer. They happen where transcription quality, reasoning quality, tool use, and conversational dynamics all collide at once. A voice agent that transcribes perfectly can still hallucinate a policy, mis-route a transfer, or talk over a user who is correcting a mistake.

Before τ-Voice, the closest evaluation options were text-based task benchmarks (like τ²-bench) that stripped out audio entirely, or turn-taking benchmarks (like Full-Duplex-Bench) that measured conversational dynamics on synthetic tasks with no real tool calls. Neither captures what actually breaks in production, which is the combination of all three failure sources running simultaneously.

τ-Voice, from Sierra and Princeton, puts grounded task completion, realistic audio, and full-duplex interaction into a single evaluation. It runs 278 tasks across three customer-service domains, with noise, accents, and interruptions in its realistic setting.

The number to know: GPT-5 with reasoning completes 85% of the same tasks in text, and GPT-4.1 without reasoning completes 54%. The best voice agent scores 51% under clean audio and 38% under realistic conditions. That is 47 points below the reasoning text baseline and 16 below the non-reasoning one, under the conditions your actual users are calling from.


What τ-Voice actually measures

The benchmark covers 278 tasks across three customer-service domains: retail (114 tasks: returns, exchanges, cancellations, order modifications), airline (50 tasks: flight changes, cancellations, seat upgrades), and telecom (114 tasks: plan changes, billing inquiries, activations).

Each task ends in a verifiable database state change. The agent either processed the return correctly or it didn’t. No partial credit for sounding confident.

The evaluation runs two passes: clean (two American-accent personas) and realistic (background noise around 15 dB SNR, burst noise about once a minute, frame drops averaging ~2% from a Gilbert-Elliott channel model, muffling on 20% of utterances, 8 kHz G.711 telephony audio, and five personas with diverse accents). The simulated users speak through ElevenLabs v3, so accents are synthetic rather than recorded. This separates capability from robustness. The gap between the two passes tells you how much your agent relies on ideal conditions.

τ-Voice also tracks four interaction-quality dimensions:

Dimension What it measures Direction
Latency Time from end of user turn to start of agent response Lower
Responsiveness Share of user turns that receive an agent response Higher
Interrupt rate How often the agent talks over the user mid-utterance Lower
Selectivity How often the agent correctly ignores backchannels and filler Higher

These four dimensions sit in tension. An agent optimized for low latency tends to respond before users finish, which drives up interrupt rate. An agent that maximizes selectivity may miss genuine starts and drop responsiveness. τ-Voice makes this tradeoff visible rather than collapsing it into a single score.

The voice user simulator runs in 200ms ticks and checks every two seconds whether to interrupt or backchannel. In the realistic setting it injects out-of-turn speech and vocal sounds (“hold on,” coughs) at about 0.7 events per minute. The tick-based architecture makes runs reproducible, which matters when comparing agent versions over time.


Where current leading voice agents fail

Three production voice agents were evaluated: OpenAI gpt-realtime-1.5 (released February 2026), Google gemini-live-2.5-flash-native-audio (December 2025), and xAI grok-voice-agent (December 2025).

Task completion (Pass@1)

Agent Clean Realistic Drop
xAI grok-voice-agent 51% 38% -13 pp
OpenAI gpt-realtime-1.5 49% 35% -14 pp
Google gemini-live-2.5-flash-native-audio 31% 26% -5 pp
Text: GPT-5 (reasoning) 85% n/a n/a
Text: GPT-4.1 (non-reasoning) 54% n/a n/a

Interaction quality (realistic condition)

Metric             OpenAI    Google    xAI
─────────────────────────────────────────
Latency (s)         0.90      1.14     1.15
Responsiveness      100%      69%      83%
Interrupt rate       14%      21%      84%
Selectivity           6%      54%      57%

The paper’s annotators reviewed 91 failed simulations from two cohorts and attributed 79% and 90% of the failures to agent behavior rather than simulator artifacts.

Logical errors ran 13 and 16 per cohort. Agents fail at reasoning even when transcription succeeds: misapplied policies, skipped tool calls, confirmed actions they hadn’t completed. Transcription errors ran 10 and 16, and the paper calls authentication the dominant bottleneck in both cohorts: agents failed to transcribe names and emails even when users spelled them letter by letter. When the agent mishears a name or email address, everything downstream fails because it cannot pull the correct account. Hallucination errors (6 per cohort) included agents verbally confirming task completion without making the tool call, with the database showing no change.

The accent ablation (retail domain) gives the sharpest single finding. xAI drops 18 percentage points in task completion with non-native accents, OpenAI drops 11, and Google drops 1. For xAI, that means nearly one in five accented callers’ tasks fail that would otherwise succeed.

The interaction quality tradeoffs tell a different story for each provider. OpenAI gets 100% responsiveness at 0.90s latency and the lowest interrupt rate (14%), but only 6% selectivity, meaning it responds to almost every backchannel and filler as if it were a new turn. xAI reaches 57% selectivity but interrupts on 84% of turns. Google balances selectivity (54%) and interruptions (21%) but misses almost one in three user turns (69% responsiveness).

No agent in this cohort is simultaneously fast, responsive, selective, and non-interruptive. These axes pull against each other, the field has not resolved the tradeoff, and τ-Voice makes it visible for each agent.


A voice agent testing checklist you can apply before shipping

The τ-Voice methodology translates directly into a pre-launch testing protocol. You don’t need to run the full benchmark to apply its logic.

1. Test authentication with real ambiguous names and emails

Authentication is the highest-risk transcription point. Feed your agent names with common pronunciation ambiguities (Stefan vs. Steven, Nguyen, O’Brien), email addresses with unusual domains, and confirmation codes with similar-sounding characters (B vs. P, M vs. N). Measure what share of sessions fail at account lookup versus later steps. If failures concentrate here, the problem is transcription, not reasoning.

2. Run with background noise before you run without it

A quiet-room test tells you nothing about production. Test with SNR around 15 dB. If task completion drops more than 20 points relative to your clean baseline, your audio pipeline is brittle. Fix that before tuning prompts.

3. Test with three accent profiles

Native speaker, moderate non-native accent, strong non-native accent. Use real speakers where you can; τ-Voice’s accents are synthetic, and synthetic audio tends to understate real-world failure. The gap between your clean and accented scores is your accent penalty. An 18-point penalty means nearly one in five of those calls fails that would otherwise succeed.

4. Measure selectivity

Inject filler turns (“uh-huh,” “okay,” “right”) at natural conversation points. Count how often your agent treats them as new intents. Below 50%, your agent will fragment conversations and generate spurious tool calls.

5. Watch for interruptions on correction turns

Users correct themselves mid-sentence constantly: “I want to change the flight, actually cancel it.” Does your agent wait for the correction or respond to the initial fragment? A high interrupt rate on correction turns means users spend energy fighting the agent rather than completing their task.

6. Verify tool calls happened

For every session where the agent says a task is done, check the database. τ-Voice found agents that verbally confirmed completions without making tool calls. A post-session audit catches this class of hallucination before users see it.

7. Stress-test multi-step policy sequencing

Real customer-service tasks have ordering requirements: verify identity before discussing account details, confirm cancellation before processing a refund. Build a small set of tasks with two or three required policy steps in sequence. Logical errors (13 and 16 per cohort in τ-Voice) often show up as policy shortcuts taken under conversational pressure. Enforce the ordering in code, not only in the prompt (see security for voicebots).


Benchmark coverage comparison

Capability                   ASR bench   Turn-taking   τ-Voice
────────────────────────────────────────────────────────────────
Transcription accuracy          yes          no           yes
Turn-taking dynamics            no           yes          yes
Grounded task completion        no           no           yes
Policy adherence                no           no           yes
Realistic audio conditions      no           partial      yes
Accent diversity                partial      no           yes
Verifiable database state       no           no           yes

FAQ

What is τ-Voice? τ-Voice (arXiv:2603.13686, Sierra and Princeton) evaluates full-duplex voice agents on grounded real-world tasks across retail, airline, and telecom domains. It combines verifiable task completion, realistic audio, and simultaneous speech handling in a single framework. It extends τ²-bench, Sierra’s earlier text-based agent benchmark.

How do current voice agents score? OpenAI gpt-realtime-1.5, Google gemini-live-2.5-flash-native-audio, and xAI grok-voice-agent score 31-51% under clean conditions and 26-38% under realistic audio. On identical tasks in text, GPT-5 with reasoning scores 85% and GPT-4.1 without reasoning scores 54%. Under realistic conditions, no voice agent reaches half of the reasoning text baseline.

What does τ-Voice measure that ASR benchmarks don’t? ASR benchmarks measure transcription accuracy on a single audio segment. τ-Voice measures whether the agent completes a real task across a full multi-turn voice conversation, with noise, accents, interruptions, and backchannels in the mix. Measuring a component versus measuring the system.

Why does authentication fail so often? Authentication requires users to speak names, email addresses, or confirmation codes. A single transcription error cascades: the agent cannot pull the account, so every subsequent tool call fails. τ-Voice’s failure analysis calls authentication the dominant bottleneck: agents misheard names and emails even when they were spelled letter by letter. This is why authentication is the right place to start stress-testing.

What is the voice gap? The performance difference between equivalent text and voice agents on the same tasks. On τ-Voice, the best voice agent (xAI, 38% realistic) reaches about 45% of the reasoning text baseline (GPT-5, 85%), or about 60% under clean conditions. Against the non-reasoning GPT-4.1 (54%), it reaches about 70%. The voice gap is the central thing τ-Voice is designed to track over time as the field improves.


Related posts:


Sources: