Microsoft VibeVoice: how 7.5 Hz tokenization makes 90-minute multi-speaker TTS tractable
TL;DR: VibeVoice (Microsoft, MIT license, August 2025) generates up to 90 minutes of audio with up to 4 speakers. The 7B model scored MOS 3.76 and 1.29% WER on the paper’s podcast test set. The key idea is a pair of continuous speech tokenizers running at 7.5 Hz, a fraction of the 25 to 75 Hz of common neural codecs, which keeps 90 minutes of audio inside a 64K-token context. Frame rate is the core constraint in long-form TTS, and that explains what VibeVoice changed. One caveat up front: Microsoft removed the TTS code from its GitHub repository in September 2025 after misuse.

Most TTS systems are good for a sentence. Some handle a paragraph. A few manage a few minutes before speaker identity drifts, prosody flattens, or coherence breaks down. VibeVoice targets 90 minutes in a single pass. Getting there took a different approach to the representation problem in speech synthesis, not only more compute or a larger model.
Why long-form TTS breaks
Every neural TTS system converts text into some intermediate token sequence before generating audio. The frame rate of that token sequence (how many tokens represent each second of audio) determines how long a sequence you can process in a single pass.
Common neural codecs like Meta’s Encodec and DAC operate at 25 to 75 frames per second, often with several codebooks per frame. At 50 Hz, one minute of audio is 3,000 frames and a 90-minute podcast is 270,000, far beyond the context most TTS language models are trained on. In practice, most autoregressive TTS systems are used for seconds to a few minutes at a time, with longer content generated in chunks.
Workarounds exist: chunk the audio into segments and stitch them, use hierarchical generation, or apply sliding window attention. Each approach introduces boundary artifacts, prosody discontinuities, or speaker drift at segment boundaries. The stitching seam is audible in most long-form TTS output.
VibeVoice solves this at the representation level rather than the generation level.
The 7.5 Hz tokenizer: what it does and why it works
The central idea in VibeVoice (arXiv:2508.19205) is a pair of continuous speech tokenizers (one acoustic, one semantic), both operating at 7.5 frames per second. The acoustic encoder has seven stages, and six downsampling layers give a cumulative 3,200x reduction from 24 kHz input.
At 7.5 Hz, 90 minutes of audio becomes about 40,500 frames, which fits in the model’s 64K-token context alongside the text. The paper reports 80x better data compression than Encodec with comparable reconstruction quality.
The two tokenizers serve different roles:
24kHz audio input
│
▼
┌──────────────────────────────────────────┐
│ Acoustic Tokenizer (σ-VAE) │
│ 7 stages → 7.5 frames/sec │
│ Captures: timbre, prosody, voice char │
│ ~340M params each (encoder, decoder) │
└──────────────────────┬───────────────────┘
│
┌──────────────────────▼───────────────────┐
│ Semantic Tokenizer (ASR-trained mirror) │
│ 7.5 frames/sec │
│ Captures: linguistic content │
│ Encoder only at inference │
└──────────────────────┬───────────────────┘
│
▼
┌──────────────────────────────────────────┐
│ LLM (Qwen2.5-1.5B or Qwen2.5-7B) │
│ Understands dialogue flow │
│ │
│ + Diffusion head │
│ Generates acoustic details │
└──────────────────────────────────────────┘
The acoustic tokenizer uses a σ-VAE architecture with mirror-symmetric encoder-decoder blocks. The semantic tokenizer mirrors the encoder’s hierarchy without the VAE component and is trained with an ASR proxy task, giving it linguistic grounding; its decoder is discarded after pre-training. Both tokenizers are frozen during LLM training; only the language model and a 4-layer diffusion head are trained.
Each 7.5 Hz frame is a continuous latent vector, not a discrete code, so one frame can carry far more detail than a single codebook entry. For each frame, the LLM’s hidden state conditions the diffusion head, which generates the acoustic latent in 10 denoising steps; the acoustic decoder then turns the latents back into a 24 kHz waveform.
What the numbers say
VibeVoice ships three model variants targeting different deployment scenarios:
| Model | MOS (avg) | WER (Whisper / NeMo) | Speaker sim. | Notes |
|---|---|---|---|---|
| VibeVoice-1.5B | 3.54 ± 0.96 | 1.11% / 1.82% | 0.548 | Qwen2.5-1.5B backbone |
| VibeVoice-7B | 3.76 ± 0.93 | 1.29% / 1.95% | 0.692 | Qwen2.5-7B backbone |
| VibeVoice-Realtime-0.5B | not reported | not reported | not reported | ~300 ms first audible audio (Microsoft’s claim) |
The test set is eight long conversational transcripts totalling about one hour, and WER is measured by running ASR (Whisper or NeMo) on the generated audio. MOS is the average of realism, richness and preference ratings. The paper compares against Dia, MoonCast, Sesame CSM, Higgs Audio V2, ElevenLabs v3 alpha and Gemini 2.5 Pro preview TTS. Note that the 1.5B model has the lower WER, while the 7B model wins on human ratings and speaker similarity.
For comparison, MOSS-TTSD v1.0, released in February 2026, supports up to 60 minutes and up to 5 speakers in a single pass. Both systems use a language model over speech tokens to keep the dialogue coherent across the session.
How the model maintains speaker identity at 90 minutes
Speaker consistency in long-form TTS is a harder problem than it looks. Short-context models can clone a voice from a 3-second reference. Maintaining that identity across 5,400 seconds of generated audio, through emotional variation and repeated hand-offs between speakers, requires something the reference alone cannot provide.
VibeVoice’s approach is to train the LLM component on long-form dialogue with a context-length curriculum: training sequences grow progressively from 4,096 to 65,536 tokens.
The LLM backbone carries the semantic coherence across the full session. It understands which speaker is active, what topic is being discussed, and what prosodic register fits the moment. The diffusion head generates the acoustic realization of each token. The combination allows speaker identity to persist not as a static template but as a dynamic character that behaves consistently across context.
The model supports up to 4 speakers per session, and they take turns rather than overlapping. For most content production (podcasts, audiobooks, educational content), turn-taking is the right model; for phone-call simulation or debate generation, you would want overlapping speech, which VibeVoice does not model.
What this enables in practice
Three content categories become tractable with 90-minute coherent multi-speaker TTS:
Full-length podcasts. A two-host, 60-minute episode can be generated in a single pass with consistent voice identities and turn-taking, instead of stitching shorter segments with audible boundaries.
Audiobooks with multiple characters. A 300-page novel narrated with distinct, consistent character voices across 8 hours of audio requires chunked generation regardless. VibeVoice’s 90-minute coherence window can cover most chapters in a single pass, which avoids stitching seams within a chapter.
Long voice monologues. For voice products that deliver extended explanations (financial summaries, technical walkthroughs), keeping speaker identity consistent across a 5 to 10 minute monologue is non-trivial. The paper does not report real-time factors, so benchmark throughput yourself.
The Realtime-0.5B variant, which Microsoft says reaches about 300 ms to first audible audio, brings VibeVoice closer to interactive use. It is still slower than the sub-100ms TTS systems: Cartesia quotes 40 ms for Sonic Turbo and ElevenLabs about 75 ms for Flash.
The open-source dimension
VibeVoice was released under the MIT license with three sizes in the family: the 0.5B Realtime variant for interactive use, the 1.5B for batch generation, and the 7B for quality-first podcast and audiobook production.
Availability changed after launch. On September 5, 2025, Microsoft removed the VibeVoice-TTS code from its GitHub repository, saying it had found uses inconsistent with the stated intent. When this post was last updated, the repository README still linked Hugging Face weights for VibeVoice-TTS-1.5B and the Realtime-0.5B model, while the TTS quick-try was disabled; the repository’s focus had shifted to VibeVoice-ASR models. Check the current state of the code, weights and terms before building a product on it. Restricting capable open voice-cloning models after misuse is a pattern worth expecting, and it is a supply risk for anyone who depends on them.
For researchers, the frozen tokenizer design is useful: the acoustic and semantic tokenizers can be reused as feature extractors without retraining. The separation of dialogue modeling (LLM) from acoustic generation (diffusion head) also means the backbone could be swapped for a newer base model, though that requires retraining the LLM and head.
Key takeaways
- VibeVoice’s 7.5 Hz tokenizers reduce a 90-minute audio stream to about 40,500 frames, which fits in a 64K-token context and makes single-pass long-form generation tractable.
- On the paper’s podcast test set (about an hour of audio), the 7B model scored MOS 3.76 and 1.29% WER; the 1.5B scored 1.11% WER. The paper does not report real-time factors.
- Up to 4 speakers per session, sequential turn-taking, up to 90 minutes: a good fit for podcast and audiobook production.
- The Realtime-0.5B variant (about 300 ms to first audio, per Microsoft) moves the family toward interactive use, trading quality for speed.
- Microsoft removed the TTS code from GitHub in September 2025 after misuse. Check what is still distributed, and under what terms, before you depend on it.
FAQ
What is Microsoft VibeVoice? VibeVoice is an MIT-licensed TTS system from Microsoft that generates up to 90 minutes of audio with up to 4 speakers. A Qwen2.5 language model (1.5B or 7B) handles the dialogue context, a small diffusion head generates the acoustic latents, and two continuous speech tokenizers run at 7.5 Hz. In the paper’s human evaluation, VibeVoice-1.5B averaged MOS 3.54 and VibeVoice-7B 3.76. Microsoft removed the TTS code from its GitHub repository in September 2025 after misuse.
What does 7.5 Hz frame rate mean for TTS? Common neural audio codecs produce 25 to 75 frames per second of audio. VibeVoice’s tokenizers produce 7.5 frames per second, through 3,200x downsampling of 24 kHz input. The paper describes this as 80x better compression than Encodec. A 90-minute clip becomes about 40,500 frames instead of 270,000 at 50 Hz, which fits in a 64K-token context.
How does VibeVoice compare to other long-form TTS systems? VibeVoice targets up to 90 minutes and 4 speakers in one pass. MOSS-TTSD v1.0 (February 2026) supports up to 60 minutes and 5 speakers. Parler-TTS and VALL-E 2 focus on short utterances and voice cloning rather than long-form dialogue. On the paper’s podcast test set (eight long conversations, about an hour of audio in total), VibeVoice-7B scored 1.29% WER with a Whisper-based judge.
What is the VibeVoice Realtime variant? VibeVoice-Realtime-0.5B is a smaller streaming model in the VibeVoice family, which Microsoft says reaches about 300 ms to first audible audio. It trades some quality for real-time capability, making it a candidate for conversational use where latency matters more than maximum fidelity.
What use cases does 90-minute coherent multi-speaker TTS unlock? Full-length podcast episodes with multiple hosts generated in one pass, audiobook chapters with consistent character voices, long-form educational content, and voice agent monologues that maintain consistent speaker identity. Previous systems required stitching shorter segments, producing prosody discontinuities and speaker drift at boundaries.
Further reading
- VibeVoice Technical Report: full architecture, training details, and benchmark results
- microsoft/VibeVoice on GitHub: MIT license; see the README for the September 2025 removal of the TTS code and current model availability
- Long-form TTS is broken: how TADA and Borderless approach the same problem
- The sub-100ms TTS race: latency benchmarks for real-time TTS systems, complementary context for voice agent deployments