Voice agent review checklist

This is the checklist I work through when I review a voice agent before or after launch. It is free to use: run it on your own system, and if a section raises questions you cannot answer, that is usually where the problems are.

Each item is a question. A good answer is a number you measured, not a belief.

1. Latency budget

Callers notice delays quickly: in ordinary conversation, people tend to start replying within a few hundred milliseconds. Measure the full path, not individual components.

Do you measure end of caller speech to first byte of agent audio on real calls, at p50 and p90?
Do you know how that total splits across turn detection, speech recognition, the LLM’s time to first token, tool calls, speech synthesis and the network or telephony hops?
How long does turn detection wait in silence before deciding the caller has finished? Is that value tuned on your traffic, or left at a default?
Does the LLM start on streaming partial transcripts, or wait for the final one?
Does speech synthesis stream audio as soon as the first sentence is ready, or wait for the full response?
Are slow tool calls (database lookups, CRM, payments) kept off the critical path, or covered with a short spoken acknowledgement?
Are the speech, LLM and telephony services in the same region as each other and as most callers?

2. Speech recognition

Is accuracy measured on your own audio, with the accents, background noise and microphones your callers actually have, not only on public benchmarks?
For phone calls, is it measured on 8 kHz narrowband audio, which loses much of the detail benchmark recordings keep?
Do callers switch languages mid-sentence? If so, is that tested?
Are names, product terms and numbers (order IDs, amounts, dates) checked separately? They often fail while the overall word error rate looks fine.
Is the transcript normalized the same way in tests and in production, so the error rate you report means something?
Is keyword boosting or a custom vocabulary used for domain terms, and is its effect measured?

3. Speech synthesis

What is the time to first audio, at p90, under real load?
Are numbers, dates, currency, abbreviations and email addresses read out correctly?
Does the voice handle the languages and names your callers use?
Is the cost per minute known at your projected volume, and compared against self-hosting?

4. Turn-taking and interruptions

Can the caller interrupt (barge in), and does the agent stop speaking quickly when they do?
How often does the agent cut the caller off mid-sentence? Is that rate measured?
Do short acknowledgements from the caller (“okay”, “mm-hm”) wrongly stop the agent?
After an interruption, does the agent know which part of its answer the caller actually heard?

5. The agent itself

Is every tool call logged with its inputs, outputs and timing, per turn?
What happens when a tool times out or fails? Does the caller hear something sensible?
Is there a clean handoff to a human, with context passed along?
Are prompts and model versions pinned, so a provider update cannot change behavior silently?

6. Cost per minute

Do you know the all-in cost per minute of conversation: telephony, recognition, LLM tokens, synthesis and infrastructure?
What does it become at ten times today’s call volume?
Which component dominates, and is there a cheaper option for the easy turns (a smaller model, cached answers)?

7. Evaluation

Is there a fixed regression set of real or realistic calls that runs before every change to prompts, models or vendors?
Are task success, latency and cost tracked together, so a change that improves one and hurts another is visible?
Are failed and abandoned calls reviewed regularly, with findings fed back into the regression set?

8. Security and privacy

A voice agent that can act on a caller’s behalf is a target. These are the questions an attacker will test for you if you do not.

Can a caller talk the agent into ignoring its instructions (prompt injection through speech)? Has anyone tried?
Which tools can the agent call, with what permissions? Does each tool have only the access its task needs?
Do irreversible actions (refunds, account changes, payments) require verification or human approval?
Is identity verified with something stronger than the caller’s voice or caller ID? Both can be faked.
Can content the agent reads (emails, tickets, web pages) carry instructions that change what it does?
Are recordings and transcripts, which often contain personal data, stored, retained and deleted according to a written policy?
Do callers hear the recording and AI disclosures your jurisdictions require?

How to use this

Go through it with the people who built the agent. Mark each item green (measured and fine), amber (unknown) or red (known problem). Amber items deserve as much attention as red ones, because you cannot fix what you are not measuring.