Speech Technology
A voice agent has about a second to reply before the caller starts to wonder if anyone is there. In that second it has to notice the caller has stopped talking, turn the audio into text, decide what to do and start speaking, often over a bad phone line and in an accent no benchmark included.
I have worked on this for more than ten years: text-to-speech for 13 Indian languages at IIT Madras, speech recognition and synthesis that shipped on more than 200 million Galaxy devices at Samsung, and voice agents that answer customer calls. Here I write about what holds up on real calls: picking ASR and TTS models, why benchmark accuracy falls apart in production, what self-hosting really costs, and how voice agents get attacked.
Start here
- ASR is solved (on benchmarks): the real-world gap every voice agent team hits: why LibriSpeech-style WER says little about real calls, and how to build an ASR test that does.
- Whisper large-v3-turbo vs Parakeet: how to choose, and why my laptop runs Whisper: accuracy, speed, language coverage and hardware support for the two most common open ASR models, with a decision tree.
- When self-hosted TTS beats the API: the 2026 cost cliff: the break-even arithmetic for self-hosting TTS against API pricing, and what breaks when you do it.
- The sub-100ms TTS race: TADA, VoXtream2, and why latency is the new quality: what TTS latency numbers actually measure, and what moves the bottleneck once TTS is fast.
- Full-duplex voice agents break on disfluency, and the benchmark finally says how: why self-corrections and multi-step tool calls break today’s voice agents, and a pre-launch test plan.
- τ-Voice benchmark: what full-duplex voice agents actually get wrong: how leading voice agents score on real customer-service tasks, and a checklist drawn from their failures.
- Security for Voicebots: Red Teaming and Blue Teaming Production Voice Agents: the attack surface of a voicebot that can take actions, and the controls that hold up in production.
Voice agents and conversational AI
- Architecting Conversational AI Systems
- WildASR: why clean benchmarks and synthetic test audio overstate ASR accuracy
- Dialog State Tracking (DST)
- Speech Boundary Detection: VAD, Forced Alignment and Endpointing
Speech recognition (ASR)
- Streaming ASR Architecture
- Multi-Speaker ASR
- ASR Beam Search Implementation
- Automatic Speech Recognition (ASR) Decoding
- Custom Language Modeling
- Cross-Lingual Speech Transfer
- Sequence-to-Sequence Speech Models
- End-to-End Speech Model Design
- Phonetic Search in Speech
- Voice Search Ranking
- Acoustic pattern matching: DTW, embeddings and hybrid audio search
Text-to-speech (TTS)
- Text-to-Speech (TTS) System Fundamentals
- Long-form TTS is broken: how TADA and Borderless are fixing it
- Voxtral TTS: what Mistral’s 3-second voice cloning delivers, and when to use it
- Microsoft VibeVoice: how 7.5 Hz tokenization makes 90-minute multi-speaker TTS tractable
- Llasa: what happens when you apply inference-time compute to speech synthesis
- MamTra: the Mamba-Transformer hybrid that cuts TTS VRAM by up to 34%
- Voice Conversion
Audio front end and signal processing
- Audio Feature Extraction for Speech ML
- Voice Activity Detection (VAD)
- Audio Preprocessing & Signal Processing
- Voice Enhancement & Noise Reduction
- Speech Separation
- Real-time Audio Segmentation
- Audio Augmentation Techniques
- Audio Quality Validation
Speakers, keywords and classification
- Speech Command Classification
- Speaker Recognition & Verification
- Real-time Keyword Spotting
- Speaker Clustering (Diarization)
- Hierarchical Speech Classification: Routing Voice Commands by Domain and Intent
- Social Voice Networks: Diarization, Moderation and Room Recommendation for Live Audio
- Speech Emotion Recognition
Production systems, cost and privacy
- Streaming Speech Processing Pipeline
- Scaling Speech Infrastructure: From Labs to Billions
- Multi-tier Speech Caching Architecture
- Cost-efficient Speech Systems
- Compute Allocation for Speech Models
- Multi-model Speech Ensemble
- Batch Speech Processing
- Speech Pipeline Orchestration
- Speech Pipeline Dependencies: Chaining VAD, Diarization and ASR in Real Time
- Multi-region Speech Deployment
- Speech Model Export
- Privacy-preserving Speech
Training, evaluation and MLOps
- Speech Tokenization
- Multi-task Speech Learning
- Distributed Speech Training
- Speech Hyperparameter Tuning
- Speech Architecture Search
- Adaptive Speech Models
- Speech Experiment Management
- Speech Quality Monitoring
- Speech Anomaly Detection
Planning a voice agent, or choosing between ASR, TTS and speech-to-speech stacks? Arun runs voice agent architecture reviews covering model choice, latency and cost. If your agent takes actions on calls, the agent security review covers how it can be abused. To build it with you, he takes on contract engineering, and for speech research problems, contract research and collaborations.
Recently added
- Whisper large-v3-turbo vs Parakeet: how to choose, and why my laptop runs Whisper
- Full-duplex voice agents break on disfluency, and the benchmark finally says how
- WildASR: why clean benchmarks and synthetic test audio overstate ASR accuracy
- Microsoft VibeVoice: how 7.5 Hz tokenization makes 90-minute multi-speaker TTS tractable
- Llasa: what happens when you apply inference-time compute to speech synthesis
Posts on new research summarize other people’s papers and link to the originals. Some posts are drafted with AI assistance and edited by Arun. If you spot an error, let me know and it gets fixed.