12 minute read

“A voice assistant is more than a speech recognizer attached to a search engine. It is a stateful entity that must navigate the social nuances of human turn-taking and intent.”

TL;DR

Voice assistants have gone through three architectures. The modular pipeline chains ASR, NLU, dialog state tracking, dialog policy and TTS. The LLM cascade replaces the middle three with a language model that calls tools, between streaming ASR and streaming TTS. Native speech-to-speech models (OpenAI’s gpt-realtime, Google’s Gemini Live models, Kyutai’s Moshi) take audio in and produce audio out in one model. Whichever you choose, the same problems decide whether it feels natural: a tight latency budget, barge-in handling with acoustic echo cancellation, contextual ASR, and a deterministic state machine for the transactional steps. For the voice agent security dimension, see the security for voicebots post. For the speech-to-speech architecture in depth, see speech-to-speech agents.

A mission control room with multiple screens showing different telemetry feeds arranged in a semicircle

1. Introduction: Beyond Single-Shot Commands

The mass-market voice assistants of roughly 2011-2018 (Siri, Alexa, Google Assistant) were built around Single-Shot Commands:

  • “Hey Google, turn on the lights.”
  • “Alexa, what’s different about a sloth and a koala?”

The technical architecture was simple: Wake Word → ASR → NLU → Action → Forget. The system kept little or no memory, so every interaction started from scratch.

Multi-turn dialog is not new. Frame-based dialog systems go back decades, and dialog state tracking has had its own research challenges since 2013. What changed with large language models is that multi-turn conversation became the default rather than a carefully scripted exception. A modern system has to maintain Dialog State across minutes of conversation, handle interruptions (Barge-In), corrections (“No, I meant Boston, UK”), and anaphora (“Send it to him”).

In this post, we look at the three architectures in use today, then at the parts every one of them needs: the Dialog State, Full-Duplex Audio, and the Latency Budget.

2. Three Architectures

2.1 The classic modular pipeline

  1. ASR (Acoustic-to-Text): Translating vibrations into a list of candidate sentences.
  2. NLU (Intent & Entity): Decoding the user’s meaning and parameters.
  3. DST (Dialog State Tracking): The “Short-term Memory” of the current flow.
  4. DP (Dialog Policy): The “Decision Maker” that decides the next system action.
  5. NLG & TTS (Text-to-Speech): Generating a natural response and speaking it.

Each module is separately trained and testable, and the behavior is predictable. The cost is design effort: every intent, slot and transition is hand-built, and users who stray from the designed paths hit dead ends.

2.2 The LLM cascade

Most voice agents built since 2023 keep the audio ends and replace the middle:

Streaming ASR → LLM with tool calling → Streaming TTS

The LLM handles NLU, state tracking, policy and response generation in one step, guided by a system prompt and a set of tools (look up an order, book a slot, transfer to a human). The state that matters for business logic, such as verified identity, order IDs and confirmed slots, should still live in code, not only in the model’s context. Frameworks such as LiveKit Agents and Pipecat orchestrate this pipeline (see voice agent frameworks).

Strengths: you get a transcript of every turn, you can swap the ASR, LLM or TTS independently, and guardrails and logging sit at clear text boundaries. Weaknesses: each hop adds latency, and the LLM never hears tone, hesitation or emotion, only text.

2.3 Native speech-to-speech models

A single model takes audio in and produces audio out:

  • OpenAI gpt-realtime: the Realtime API launched in beta in October 2024 and reached general availability with the gpt-realtime model on August 28, 2025, adding SIP phone calling and remote MCP tool servers.
  • Google Gemini Live: Google’s Live API serves native-audio Gemini models for streaming, interruptible voice conversations.
  • Kyutai Moshi (September 2024): an open full-duplex speech-text model that listens and speaks at the same time, with a theoretical latency of 160 ms and about 200 ms in practice (arXiv:2410.00037).

Strengths: lower latency, natural interruptions and backchannels, and prosody that reflects what the model heard. Weaknesses: less control over each stage, harder auditing (you may not get a reliable transcript of what the model “heard”), and tool use under realistic conditions is still weak. On the τ-Voice benchmark, the best tested voice agent completed 38% of customer-service tasks under realistic audio, and on Full-Duplex-Bench-v3, no system passed more than 60% of disfluent tool-use scenarios.

  Modular pipeline LLM cascade Speech-to-speech
Latency Medium Highest (three hops) Lowest
Control and auditability Highest High (text at every boundary) Lowest
Handles open-ended speech Poorly Well Well
Hears tone and emotion No No Yes
Best for Narrow, regulated flows Most tool-using agents today Conversational feel, low latency

3. High-Level Architecture: The Latency Budget

People leave about 200 ms between turns on average in natural conversation (Stivers et al., 2009, PNAS). A voice agent that answers within roughly 500-800 ms feels responsive; past a second it feels hesitant, and past two seconds it feels broken.

A typical budget for an LLM cascade:

Stage Latency Budget
End-of-turn detection + final ASR 200-300ms
LLM time to first token 200-400ms
TTS first audio 50-100ms
Total (no tool call) 450-800ms
External API or tool call, when needed +300ms or more

To achieve this, we use Streaming Orchestration. ASR streams partial transcripts while the user is still speaking, the LLM starts as soon as the turn ends, and TTS starts speaking the first sentence fragment while the LLM is still generating the rest. Speech-to-speech models collapse the three middle rows into one, which is where most of their latency advantage comes from. For which TTS systems fit this budget, see the sub-100ms TTS race.

4. Implementation: The Dialog State Machine

In complex domains like flight booking, simple “intent matching” fails. We use a Frame-based State Machine.

class DialogManager:
    def __init__(self):
        self.state = "IDLE"
        self.slots = {"destination": None, "date": None}

    def process_turn(self, nlu_result):
        # 1. Update Slots from NLU
        for entity in nlu_result.entities:
            if entity.type in self.slots:
                self.slots[entity.type] = entity.value

        # 2. Transition Logic (The State Machine)
        if not self.slots['destination']:
            reply, self.state = "Where are you flying to?", "AWAIT_DESTINATION"
        elif not self.slots['date']:
            reply, self.state = f"Understood. When do you want to fly to {self.slots['destination']}?", "AWAIT_DATE"
        else:
            reply, self.state = "Everything is set. Booking now.", "FINISHED"
        return reply, self.state

The same frame survives in LLM-based agents. Instead of an NLU model filling slots, the LLM proposes slot values through a structured tool call (for example set_trip(destination="Boston", date="2026-03-14")), and code validates them, stores them, and decides whether the booking can proceed. The LLM handles the conversation; the state machine still owns the transaction.

5. Advanced: Handling “Speech Interruptions” (Barge-in)

How does a system know when to stop talking if the user interrupts?

  • Full-Duplex Audio: The system is always “Listening” while “Speaking.”
  • Acoustic Echo Cancellation (AEC): The system must “subtract” its own voice from the incoming audio stream.
  • Turn detection: Voice activity detection alone cannot tell a real interruption from a backchannel (“uh-huh”) or a cough. Agent frameworks such as LiveKit Agents and Pipecat add model-based end-of-turn detection on top of VAD.
  • Latency Connection: If interrupt detection is slow, the system feels less intelligent.

6. Real-time Implementation: Contextual ASR

The ASR model shouldn’t just be a general model. It should be State-Aware.

  • If the Dialog Manager is in the AWAIT_DATE state, the ASR’s Language Model should be weighted toward numbers and months.
  • This “Contextual Boosting” reduces the WER significantly for ambiguous words.
  • Most streaming ASR APIs expose this as keyword or phrase boosting; in an LLM cascade, the agent can update the boost list as the conversation moves between steps.

7. Comparative Analysis: Rule-based vs. LLM-based Dialog

Metric Rule-based (Rasa/Dialogflow) LLM with tool calling
Control Absolute Probabilistic
Consistency High Medium
Complexity High (Human setup) Low setup, ongoing evaluation
Best For Banking / Booking steps Open-ended support, FAQs, triage

8. Failure Modes in Conversational AI

  1. Dialog State Collapse: The user changes their mind multiple times, and the state machine gets stuck.
    • Mitigation: Implement a “Reset State” command and confidence threshold.
  2. Entity Confusion: Ambiguity in multi-entity sentences.
  3. Vocal Sarcasm: Misinterpreting positive words used sarcastically.
  4. Self-corrections mid-sentence: “Book it for Friday, no, Saturday” after the agent has already acted on Friday.
    • Mitigation: Treat tool calls as tentative until the user finishes, and confirm before committing anything with more than one mutable argument.
  5. Claimed actions that never happened: LLM agents sometimes say a task is done without making the tool call.
    • Mitigation: Check tool results in code, and only let the agent confirm what the system of record shows.

9. Example: Context Carryover Across Turns

Assistants such as Alexa added context carryover so follow-up questions work without repeating the subject:

  • If you ask “How tall is the Eiffel Tower?” and then “When was it built?”, the system performs Coreference Resolution to map “it” back to “Eiffel Tower” from the previous turn.
  • In an LLM cascade this comes almost for free, because the conversation history is in the model’s context. The engineering work moves to deciding how much history to keep, and which facts (such as verified identity) must come from the system of record rather than from the transcript.

10. Key Takeaways

  1. State is Soul: A conversation is a trajectory through a state space, even when an LLM is doing the talking.
  2. Latency is the UI: A slow voice system is a broken voice system. Budget every hop.
  3. Pick the architecture for the job: modular pipelines for narrow regulated flows, LLM cascades for most tool-using agents, speech-to-speech models when conversational feel and latency matter most.
  4. Hybridization wins: Let the LLM run the conversation, but keep transactions in deterministic code.
  5. Feedback Loops: Use data from failed turns to refine your prompts, tools and state transitions.

FAQ

What is the latency budget for a conversational AI system?

People typically leave about 200 ms between turns in conversation, so a voice agent that answers within roughly 500-800 ms feels responsive, and one that takes more than about two seconds feels broken. In an LLM cascade, a typical budget is 200-300 ms for end-of-turn detection and final ASR, 200-400 ms for the LLM’s first token, and 50-100 ms for TTS first audio, plus a few hundred milliseconds whenever a tool or external API is called. Streaming is essential: the LLM starts as soon as the transcript is final, and TTS starts on the first sentence fragment.

How does a dialog state machine work for voice assistants?

A frame-based dialog state machine maintains slots (like destination and date for a flight booking) that are filled progressively from NLU entity extraction. The state transitions determine what question to ask next based on which slots are missing, enabling multi-turn conversations that feel natural rather than requiring the user to provide everything in one utterance. In LLM-based agents the same frame still exists, but the LLM proposes slot values through structured tool calls and code validates and stores them.

What is barge-in and how do conversational AI systems handle it?

Barge-in is when a user interrupts the system while it is speaking. Full-duplex audio systems listen continuously while speaking, using Acoustic Echo Cancellation (AEC) to subtract the system’s own voice from the incoming audio stream. Fast interrupt detection, combined with ignoring backchannels like “uh-huh”, is what makes the system feel attentive.

When should you use rule-based versus LLM-based dialog management?

Rule-based flows (Rasa, Dialogflow-style state machines) offer tight control and high consistency, which suits banking and booking steps where predictability matters. LLM-based dialog handles open-ended phrasing, topic changes and long-tail questions with far less hand-built design. Most production voice agents are hybrids: the LLM runs the conversation, while authentication, payments and other transactional steps go through deterministic code and policy checks.

Should I build a cascade or use a speech-to-speech model?

A cascade (ASR, LLM, TTS) gives you transcripts, component choice, easier guardrails and audit logs. A speech-to-speech model gives lower latency, hears tone and emotion directly, and handles interruptions more naturally, at the cost of less control and harder auditing. Public benchmarks such as τ-Voice and Full-Duplex-Bench-v3 show both kinds still fail many realistic tool-use tasks, so test on your own flows.

Related posts

Voice Search Ranking

25 minute read

“Play Call Me Maybe”. Did you mean the song, the video, or the contact named ‘Maybe’?