MamTra: the Mamba-Transformer hybrid that cuts TTS VRAM by up to 34%
Picture a voice agent stack where the TTS model shares a GPU with the LLM. It runs fine until traffic grows: more concurrent sessions, longer contexts, a bigger batch. Then the TTS process runs out of memory mid-sentence, and the fix on the table is a larger GPU tier that costs far more per month.
Benchmarks show you MOS scores and WER. They don’t show you how much memory headroom a TTS model leaves for everything else on the card.
MamTra (arXiv:2603.12342, KAIST and Chung-Ang University, March 2026) is an attempt to solve exactly this problem: not by quantizing weights or pruning layers, but by rethinking which architectural primitive handles which part of the sequence.
TL;DR
MamTra replaces half the Transformer layers in a TTS language model with Mamba selective state-space layers. The result: up to 34% less inference VRAM than CosyVoice 2, NMOS dropping 0.02 to 3.66, WER rising 0.25 percentage points to 2.28%. For co-located TTS+LLM stacks on a single GPU, that margin can decide whether everything fits on one card. See TTS system fundamentals for the background on Transformer TTS architecture before diving in.

Why Transformer TTS is memory-expensive
Autoregressive Transformer TTS generates one speech token at a time, and every new token attends to every previous token. Each attention layer keeps the keys and values of all earlier tokens, so the KV cache grows linearly with sequence length, and the attention compute for each new token grows with it.
Over a whole sequence, attention compute is O(n²) in sequence length n, while the KV cache is O(n). For a sentence, this is manageable. For a long paragraph, many concurrent streams, or a multi-turn session where TTS context accumulates, memory becomes the constraint.
LLM-based TTS systems like CosyVoice 2 model speech as a sequence of discrete tokens: CosyVoice 2 uses 25 speech tokens per second, and other codec language models run at 12.5 to 75 frames per second, often with several codebooks per frame. A 30-second utterance is 750 tokens for CosyVoice 2. Every one of those tokens keeps keys and values in every attention layer, and on a GPU already shared with an LLM, that memory competes with everything else.
A typical co-located deployment (Whisper Large v3 at 5-8 GB FP16, an 11B LLM at roughly 22 GB, and a Transformer TTS model at 6+ GB) lands at 33-36 GB total, right at the A100 40GB limit before KV cache growth pushes you over. The fix is usually to either drop to a smaller LLM, add a second GPU, or accept degraded context. None of those are free.
For scale, here is the KV cache of CosyVoice 2’s language model (a Qwen2.5-0.5B backbone: 24 layers, 2 KV heads, head dimension 64, bf16). Per token it is 2 × 24 × 2 × 64 × 2 bytes = 12 KiB:
KV cache per stream (CosyVoice 2 LM, bf16)
Context length: 512 1024 2048 4096
KV cache (MiB): 6 12 24 48 <- linear in length
Per stream, that is small. It adds up with concurrency (32 streams at 2,048 tokens is 768 MiB) and with longer contexts, and the paper’s measured savings also include activations and other runtime memory. MamTra’s pitch is that full attention in every layer is not necessary: layers that do mostly local acoustic modeling can use a fixed-size recurrent state instead.
What MamTra does: where Mamba replaces attention and where it doesn’t
MamTra uses an interleaved architecture where Transformer layers handle global semantic context and Mamba selective state-space layers handle local acoustic modeling. In the 1:1 configuration, 12 of 24 layers are Mamba and 12 remain Transformer, with Transformer layers recurring periodically through the stack (“BlockBeg” placement: a Transformer layer at the start of each block).
Full Transformer TTS LM (24 layers):
T T T T T T T T T T T T T T T T T T T T T T T T
Every layer keeps a KV cache that grows with sequence length
MamTra 1:1, interleaved (schematic):
T M T M T M T M T M T M T M T M T M T M T M T M
12 layers keep a KV cache (still linear in length, half the size)
12 layers keep a fixed-size recurrent state
T = Transformer attention layer (KV cache grows with length)
M = Mamba selective state-space layer (constant-size state)
The diagram is schematic: the paper names its placement strategies but does not print a full layer map for each ratio.
Why keep any Transformer layers at all? The paper’s answer is that attention layers carry global context and semantic reasoning, while Mamba layers model local acoustic structure efficiently. The 1:11 results show what happens when too few attention layers remain: naturalness drops sharply.
The paper compared three families of placement: interleaved (BlockBeg, BlockEnd), contiguous (Front, Middle, Back, Sandwich) and data-driven (ranking layers by how much WER suffers when each is replaced). Its final models use BlockBeg for 1:1 and 1:3, and the importance ranking for 1:5 and 1:11.
Mamba’s constant memory per generation step is what creates the VRAM savings. Attention must store all previous keys and values; Mamba compresses the history into a fixed-size hidden state that updates at each step but doesn’t grow. At 2,048 context length, the paper reports per-token compute falling from 1.78 to 1.64 TFLOPs (1.4×10¹¹ FLOPs less per token) and inference memory falling by up to 34% versus CosyVoice 2 and 17% versus Zonos-v0.1, measured on an NVIDIA A6000.
Training the hybrid without starting from scratch uses three-component distillation: cross-entropy on ground-truth tokens, Skew KL divergence between teacher and student logits, and MSE on token embeddings. This transfers the Transformer teacher’s learned representations into the hybrid student, which is why MamTra reaches competitive quality training on about 0.5k hours of LibriTTS, less than 2% of the teacher’s training data.
The quality-memory tradeoff: what 34% savings costs you
MamTra 1:1 scores NMOS 3.66 vs CosyVoice 2’s 3.68, a 0.02 difference well inside the error bars, with WER rising from 2.03% to 2.28%, a 0.25 percentage point absolute increase. The quality cost is real but small at 1:1. At 1:11 it is large.
Results on the Seed-TTS English test set (from the paper; TFLOPs are per token at 2,048 context):
| Configuration | NMOS | UTMOS | Speaker sim. | WER | TFLOPs |
|---|---|---|---|---|---|
| CosyVoice 2 (teacher) | 3.68 | 4.15 | 0.66 | 2.03% | 1.78 |
| MamTra 1:1 | 3.66 | 4.16 | 0.72 | 2.28% | 1.64 |
| MamTra 1:3 | 3.65 | 4.14 | 0.72 | 3.26% | 1.57 |
| MamTra 1:5 | 3.65 | 4.13 | 0.72 | 2.53% | 1.54 |
| MamTra 1:11 | 3.27 | 4.15 | 0.72 | 3.99% | 1.52 |
| Zonos-v0.1 | 3.18 | 3.63 | 0.67 | 3.42% | 7.32 |
The memory figure (up to 34% less than CosyVoice 2, 17% less than Zonos-v0.1) is reported for MamTra 1:1. The paper does not give absolute gigabytes.
UTMOS, an automated naturalness predictor, is essentially tied across all variants. Speaker similarity is higher for every MamTra variant (0.72) than for the CosyVoice 2 teacher (0.66).
Where does quality go first? Word error rate. The 0.25 percentage point WER increase at 1:1 is the first casualty of replacing attention with Mamba. It’s small, and still well below Zonos-v0.1’s 3.42%, but it suggests the Mamba layers are slightly less precise at articulation than full attention. At 1:3, WER rises to 3.26%. At 1:5, which uses the importance-based placement instead of BlockBeg, it is 2.53%, so placement matters as much as the ratio. At 1:11, NMOS falls to 3.27 and WER to 3.99%: an audible regression.
The honest takeaway: if you’re comparing to CosyVoice 2 as your baseline, the 1:1 hybrid is close to a free lunch on naturalness. WER being 0.25 points higher matters more in high-accuracy applications (medical, legal dictation) than in casual voice agent interactions where slight imprecision is acceptable.
There’s one gap in the benchmarks worth noting: the paper reports FLOPs and VRAM savings but not wall-clock RTF (real-time factor). Whether the linear-time Mamba blocks translate to measurable latency reduction on real hardware, especially on A100s or H100s where CUDA attention kernels are highly optimized, isn’t answered here. For latency-critical applications, you’d want to benchmark RTF independently.
Deployment decision guide: when MamTra changes the calculus
The paper reports relative savings only. If your TTS process uses 6 to 8 GB, a 34% cut would free roughly 2 to 2.7 GB; measure on your own stack before planning around it. The savings matter when the GPU is already under pressure: co-located TTS+LLM stacks, high-concurrency multi-stream deployments, or constrained edge hardware. They matter less when TTS runs on a dedicated node with headroom.
flowchart TD
A[New TTS deployment] --> B{Is TTS co-located\nwith an LLM?}
B -->|Yes| C{Does TTS+LLM\nfit current GPU tier?}
B -->|No| D{Running 4+ concurrent\nTTS streams?}
C -->|No - OOM risk| E[MamTra 1:1\nHigh value ✓]
C -->|Yes - with headroom| F{How much headroom?}
F -->|< 3 GB margin| G[MamTra 1:1\nPreventive ✓]
F -->|> 3 GB margin| H[Full Transformer\nStay put]
D -->|Yes| I{Quality requirement?}
D -->|No| H
I -->|Casual / assistant| J[MamTra 1:1\nHigh value ✓]
I -->|Medical / legal| K[Evaluate WER carefully\nMay not be worth it]
J --> L[Deploy with\nBlockBeg 1:1 config]
G --> L
E --> L
Co-located TTS + LLM (the primary use case). A voice agent stack combining Whisper Large v3, an 11B LLM, and CosyVoice 2 uses roughly 33-36 GB of VRAM before KV cache growth. An A100 40GB has 4-7 GB of headroom, which disappears as conversation context grows. If the 34% saving holds on your stack, replacing CosyVoice 2 with MamTra 1:1 could recover around 2 GB, which may be the margin between out-of-memory risk and stable operation. For voice agents where long conversations are common, this can be the difference between reliable operation and intermittent failures.
Multi-stream TTS on a single GPU. If you’re running four concurrent TTS streams on an A100, 34% VRAM reduction per stream may let you run five or six instead, improving GPU utilization without adding hardware. The economics depend on your traffic patterns, but at scale the per-stream savings compound.
Edge and resource-constrained hardware. The Qwen3-TTS 1.7B model already fits on a 24 GB consumer GPU, but for higher-quality models in the 500M-1B parameter range, MamTra’s efficiency matters more at the edge than in data centers. The small distillation set (about 500 hours) also suggests that converting an existing Transformer TTS model is cheap, though the paper does not test adapting to new languages.
When to stay with full Transformer. If TTS runs on a dedicated GPU node with room to spare, MamTra’s savings don’t justify the 0.25-point WER increase. If you’re already using Q4_K_M quantization (which cuts weight memory by roughly 70% at some quality cost), MamTra adds marginal benefit. For applications requiring high articulation accuracy (medical transcription, legal dictation, audiobooks), the WER tradeoff warrants careful evaluation rather than automatic adoption.
The broader question MamTra raises is architectural. The field has been treating Transformer-based TTS as essentially fixed: tune the training data, tune the prompting, optimize the inference kernel. MamTra is a signal that the backbone architecture itself is still in play, and that selective state spaces may earn a place in production TTS rather than staying an experiment.
For the full picture of where TTS sits in 2026 (including Voxtral, ElevenLabs v3, and the emerging voice cloning landscape), see the Voxtral and the 2026 TTS landscape post. For GPU memory budgeting in the broader voice agent stack, see voice agent architecture.
Key takeaways
- MamTra 1:1 uses up to 34% less VRAM than CosyVoice 2 with a 0.02 NMOS drop and a 0.25-point WER increase: a small quality cost for a large memory gain
- The architecture is an interleaved hybrid: Transformer layers carry global context, Mamba layers handle local acoustic modeling
- Training uses three-component distillation from a Transformer teacher, reaching competitive quality with less than 2% of the teacher’s training data
- The 1:1 ratio is the deployment-safe choice: more Mamba-heavy variants cut FLOPs further, but WER rises and 1:11 loses naturalness
- The KV cache grows linearly, not quadratically: hybrids shrink it by removing attention layers, and the benefit grows with context length and concurrency
- The savings matter most for co-located stacks: dedicated TTS nodes with GPU headroom don’t benefit enough to justify the WER tradeoff
- Wall-clock RTF isn’t reported: FLOPs and VRAM savings don’t automatically translate to lower latency on optimized hardware; benchmark before assuming
FAQ
What is MamTra and how does it differ from standard Transformer TTS?
MamTra (arXiv:2603.12342, KAIST and Chung-Ang University, March 2026) replaces a portion of the Transformer layers in a TTS language model with Mamba selective state-space layers. Attention’s compute grows quadratically with sequence length and its KV cache grows linearly; a Mamba layer keeps a fixed-size state instead. The 1:1 ratio uses 12 Mamba and 12 Transformer layers from a 24-layer base, reducing inference VRAM by up to 34% vs CosyVoice 2 while keeping NMOS at 3.66 vs the teacher’s 3.68.
Does MamTra compromise audio quality?
The quality drop is small at the 1:1 ratio. On the Seed-TTS English test set, MamTra 1:1 scores NMOS 3.66 vs CosyVoice 2’s 3.68, a 0.02 difference within the error bars. WER rises from 2.03% to 2.28%, a 0.25 percentage point increase. UTMOS (4.16 vs 4.15) is a tie, and speaker similarity is higher (0.72 vs 0.66). MamTra 1:11, the most Mamba-heavy variant, degrades clearly (NMOS 3.27, WER 3.99%), making 1:1 the practical deployment choice.
How does MamTra’s knowledge distillation work?
MamTra trains with three simultaneous loss signals from a pretrained Transformer teacher: cross-entropy supervision on ground-truth tokens, Skew KL divergence between teacher and student logits, and MSE on token embeddings. This three-component approach lets MamTra reach competitive quality on roughly 0.5k hours of LibriTTS, less than 2% of the teacher’s training data, because the student inherits learned representations rather than discovering them from scratch.
When does MamTra’s 34% VRAM saving change your deployment decision?
The saving matters most in three scenarios: (1) co-located TTS+LLM stacks where the LLM already consumes most of the GPU’s VRAM, (2) multi-stream deployments where 4-8 concurrent TTS instances share a single A100, and (3) edge or cloud instances where the next GPU tier carries a significant cost jump. It matters less when TTS runs on a dedicated node or when you’re already using Q4_K_M quantization, which provides larger memory reduction at a different quality tradeoff.
What is the Mamba-Transformer ratio, and which should I use?
The ratio is Transformer layers to Mamba layers in a 24-layer model. MamTra 1:1 (12 Transformer, 12 Mamba) gives the best balance: up to 34% less VRAM with negligible quality loss. 1:3 (6 Transformer, 18 Mamba) raises WER to 3.26%. 1:5 (4 Transformer, 20 Mamba), placed by an importance ranking, holds WER at 2.53%. 1:11 (2 Transformer, 22 Mamba) has the lowest FLOPs, but NMOS falls to 3.27 and WER rises to 3.99%. For production deployments, 1:1 is the safe choice.
Further reading
- MamTra paper on arXiv (2603.12342): the primary source, with audio samples at mamtratts.github.io
- Mamba: Linear-Time Sequence Modeling with Selective State Spaces, the original Mamba paper by Gu and Dao
- TTS system fundamentals: background on Transformer TTS, KV cache mechanics, and the two-stage pipeline
- Cost-efficient speech systems: GPU memory budgeting, quantization tradeoffs, and hardware selection for production speech pipelines