Llasa: what happens when you apply inference-time compute to speech synthesis
TL;DR: Llasa (arXiv:2502.04128, HKUST, February 2025) applies inference-time compute scaling to text-to-speech: instead of taking one sample, it searches over candidates scored by speech understanding models. The backbone is a standard Llama model plus X-Codec2, a single-codebook quantizer that puts all speech information into one token stream. Search trades compute and latency for quality on whatever the verifiers measure. For practitioners, this means a new knob: how much inference budget do you want to spend per utterance? Note that the released checkpoints are CC BY-NC 4.0.

The AI field spent two years learning that scaling inference compute improves text reasoning. Reasoning models such as o1 and DeepSeek-R1 spend it on long chains of thought learned through reinforcement learning. Search is the other way to spend it: sample many candidates (best-of-N, beam search, tree search) and let a verifier pick the best.
Llasa asks: what if the same principle applies to speech?
The answer is yes, and the implications are practical enough to matter for anyone building TTS pipelines today.
What inference-time scaling actually means for TTS
Training-time scaling is familiar: more parameters, more data, longer training runs produce better models. The results transfer to every inference call uniformly: you pay once at training, benefit forever.
Inference-time scaling works differently. The model is fixed; you spend more compute at generation time to improve individual outputs. For a language model, this means sampling multiple candidate responses and selecting the best one via some scoring function. For a TTS system, the same principle applies: generate multiple candidate token sequences and select the one that best satisfies measurable speech quality criteria.
Llasa implements this as verifier-guided beam search:
Input text
│
▼
LLaMA decoder generates N candidate token continuations
│
├─── Candidate 1 → Verifier scores (ASR accuracy, speaker consistency)
├─── Candidate 2 → Verifier scores
├─── Candidate 3 → Verifier scores
│ ...
└─── Candidate N → Verifier scores
│
▼
Top-M candidates advance
│
▼
Next decoding step
The verifiers are existing speech understanding models: Whisper large-v3 for content accuracy, a WavLM-based speaker verification model for timbre, and emotion2vec for emotion similarity. In the paper’s main setup, beam search scores partial sequences over the first 2 seconds (16 beams, each extended by 16 candidates of 0.5 seconds), then best-of-N scores complete continuations for the rest.
The trade-off is explicit: more beam steps = more candidates evaluated = higher latency. The paper documents that “scaling inference-time compute shifts the sampling modes toward the preferences of specific verifiers, thereby improving emotional expressiveness, timbre consistency, and content accuracy” (arXiv:2502.04128, abstract).
The architectural simplicity that enables scaling
Most TTS systems use cascaded architectures: separate models for text-to-phoneme, phoneme-to-acoustic, acoustic-to-waveform. Each stage has its own representation, its own training objective, its own failure modes. Scaling one stage doesn’t automatically improve the others.
Llasa’s architecture collapses this into a single model:
Text input
│
▼
┌──────────────────────────────────────┐
│ LLaMA decoder (1B / 3B / 8B) │
│ Standard transformer decoder │
│ Trained jointly on text + speech │
└──────────────────┬───────────────────┘
│ Token stream (50 Hz)
▼
┌──────────────────────────────────────┐
│ X-Codec2 decoder │
│ Single-codebook FSQ, 65,536 codes │
│ 50 tokens per second │
│ 2.47% WER on LibriSpeech test-clean │
└──────────────────────────────────────┘
│
▼
Audio waveform
X-Codec2 is the enabling piece. Codecs like Encodec use multiple RVQ (residual vector quantization) layers, with each layer refining the previous one (a hierarchical, multi-stream representation that’s powerful but awkward for a plain language model). X-Codec2 instead uses a single codebook of 65,536 codes built with finite scalar quantization (FSQ), encoding content, prosody, timbre, and emotion into one discrete token stream. The paper calls it a single-layer vector quantizer.
FSQ also sidesteps codebook collapse, a known failure mode of learned VQ codebooks in which only a small subset of codes gets used. FSQ codes are a fixed grid rather than learned vectors.
Because all speech information flows through one token stream at 50 Hz, scaling decisions are straightforward: increase the LLaMA model size, increase training data, or increase inference compute. There’s no cascade to optimize across.
What the numbers say
The Llasa paper (arXiv:2502.04128, Zhen Ye, Xinfa Zhu, Chi-Min Chan, Xinsheng Wang, Xu Tan et al., HKUST) reports:
- X-Codec2 achieves UTMOS 4.13 (naturalness) and 2.47% WER on LibriSpeech test-clean reconstruction (ground truth: 4.09 and 1.96%), with speaker similarity 0.82. That is codec round-trip quality before the LLM even touches it.
- Scaling train-time compute (larger model, more data) “consistently improves the naturalness of synthesized speech and enables generation of more complex and accurate prosody patterns”
- Inference-time scaling shifts outputs toward what each verifier rewards, improving emotional expressiveness, timbre consistency and content accuracy
For deployment context:
| Model | Base LLM | bf16 weights (approx.) |
|---|---|---|
| Llasa-1B | Llama 3.2 1B | ~3 GB |
| Llasa-3B | Llama 3.2 3B | ~7 GB |
| Llasa-8B | Llama 3.1 8B | ~17 GB |
Weight sizes include the 65,536 added speech tokens; add the codec, the verifier models (Whisper large-v3 alone is about 3 GB in fp16) and the KV cache. The paper reports no real-time factors. The search setup gives a sense of cost: 16 beams each extended by 16 candidates means 256 candidate continuations scored per half-second step during the beam phase, against one for plain sampling. Expect search to cost many times the compute of a single sample.
For comparison, Kokoro-82M, a StyleTTS2-based model, runs far faster than real time on modest hardware. Cartesia Sonic Turbo and ElevenLabs Flash are optimized for sub-100ms first-chunk latency. Llasa doesn’t compete with them on speed. It competes on the quality ceiling when latency budget allows.
Where inference-time scaling breaks down for TTS
Practitioners will hit these limitations:
Verifier brittleness. The beam search quality depends entirely on the verifier’s scoring function. A Whisper-based pronunciation verifier optimizes for transcription accuracy, not naturalness. A speaker verification model optimizes for speaker identity consistency, not prosodic expressiveness. Different verifiers produce different quality improvements, and combining verifiers requires calibrating their relative weights, which the paper doesn’t prescribe.
Streaming incompatibility. Search is fundamentally non-streaming: you need to evaluate multiple candidates before committing to a path. Llasa+ (arXiv:2508.06262, August 2025) adds multi-token prediction (a reported 1.48x speedup) and a causal decoder for streaming reconstruction, but verifier-guided search over long candidates still does not fit a low-latency streaming deployment.
Emotional control without a steering mechanism. Inference-time scaling “improves emotional expressiveness” through verifiers, but doesn’t expose a knob for which emotion. StyleTTS2 and CosyVoice have explicit style vectors; Llasa’s emotional variation emerges from verifier-guided search rather than explicit direction. This works for naturalness but not for controlled emotional performance.
Pronunciation debugging is harder. In a cascaded system, mispronunciation is diagnosable: you know which stage failed. In Llasa’s unified stream, a mispronounced word could reflect model confusion, codec error, or verifier failure. Multi-layer feedback loops without per-layer diagnostics make production debugging opaque.
The practical decision point
The speech tokenization architecture underlying Llasa (unified discrete token stream, single LLM decoder) represents a clean direction for TTS research. The scaling laws are predictable: more model, more data, more inference compute all improve quality.
The practical question is whether your workload fits the constraint:
- Latency-critical voice agents (< 200ms first chunk): Use Kokoro, Cartesia, or ElevenLabs. Llasa’s beam search overhead is incompatible.
- Batch content generation (audiobooks, podcasts, e-learning): search fits well here. Its overhead costs compute minutes, not seconds of user patience.
- Quality-first single-speaker narration: Llasa-3B or 8B with verifier-guided search is worth testing against Kokoro or Parler-TTS on your hardest sentences. The paper does not run that comparison, so listen for yourself.
- Commercial products: the released checkpoints are CC BY-NC 4.0, so you would need to train your own model with the open training code or license elsewhere.
- Multilingual with code-switching: Llasa shows in-context learning. Provide a 2-3 second reference in one language and it adapts for cross-lingual generation. The 250,000-hour training corpus includes both Chinese and English.
The training code is at zhenye234/LLaSA_training. Hugging Face checkpoints are at HKUSTAudio/Llasa-1B, HKUSTAudio/Llasa-3B and HKUSTAudio/Llasa-8B under CC BY-NC 4.0, which prohibits commercial use.
Key takeaways
- Llasa applies inference-time scaling (verifier-guided beam search and best-of-N) to TTS. More inference compute improves what the verifiers measure, at the cost of compute and latency.
- X-Codec2 encodes all speech information into a single discrete token stream (50 Hz, a single 65,536-code FSQ codebook), enabling a single Llama decoder to handle the full generation without cascading.
- X-Codec2 achieves 2.47% WER and UTMOS 4.13 on LibriSpeech test-clean reconstruction; the codec itself is high-quality before the LLM contributes.
- The paper reports no real-time factors. Search scores hundreds of candidates per step in its main setup, so budget for many times the compute of plain sampling.
- Llasa is wrong for low-latency voice agents and right for batch content generation, quality-first narration, and research into speech scaling laws. The checkpoints are non-commercial.
FAQ
What is Llasa TTS? Llasa (arXiv:2502.04128, HKUST, February 2025) is an open-weight TTS system applying inference-time compute scaling to speech synthesis. A standard Llama decoder (1B/3B/8B) generates a single unified token stream via X-Codec2, with search guided by speech verifiers selecting higher-quality outputs. Available via HKUSTAudio on Hugging Face under CC BY-NC 4.0.
How does inference-time scaling work for TTS? Standard TTS samples one token sequence. Llasa samples many candidate continuations, scores them with speech verifiers (Whisper for content accuracy, speaker verification for timbre, an emotion model for expressiveness), and keeps the highest-scoring paths. More candidates mean better scores on the verifier’s metric, at higher compute and latency.
What is X-Codec2 and why does it matter? X-Codec2 is a single-codebook speech codec (FSQ, 65,536 codes, 50 Hz) that encodes content, prosody, timbre, and emotion into one discrete token stream. This unified representation lets a single LLaMA decoder handle complete TTS generation, making scaling decisions simple: scale model, data, or inference compute.
How does Llasa compare to Kokoro and CosyVoice? Kokoro-82M is the efficiency pick: fast on modest hardware and strong on quality per parameter. CosyVoice targets multilingual production TTS. Llasa targets quality-first scenarios with a flexible compute budget, where verifier-guided search improves the properties the verifiers measure.
What are the deployment options? HKUSTAudio checkpoints on Hugging Face under CC BY-NC 4.0: 1B, 3B and 8B, with bf16 weights of roughly 3, 7 and 17 GB before the codec, verifiers and KV cache. Llasa+ (arXiv:2508.06262) adds multi-token prediction for a reported 1.48x speedup and a causal decoder for streaming.
Further reading
- Llasa: Scaling Train-Time and Inference-Time Compute for Llama-based Speech Synthesis, the paper
- Llasa+ streaming and acceleration: August 2025 follow-up with multi-token prediction and streaming
- The sub-100ms TTS race: context for where Llasa sits in the latency spectrum
- TTS system fundamentals: how unified architectures like Llasa differ from cascaded systems