KV cache for MoE: why sparse experts don’t shrink the memory bill
“MoE sparsifies the computation. The memory bill arrives in full.”
TL;DR
Mixtral-8x7B activates 2 of 8 experts per token, so it computes like a 12.9B model. It does not store like one. All 46.7B parameters (about 93 GB in BF16) must stay in GPU memory, and the KV cache is set by the attention layers, which routing never touches: 128 KiB per token, exactly what dense Mistral 7B uses. The weights take a fixed bite of memory, the KV cache fills the rest, and that remainder caps your batch size. Since MoE needs large batches to keep its experts busy, the KV budget is what limits MoE serving throughput. DeepSeek’s MLA shrinks the cache in the architecture (about 70 KB per token versus 516 KB for Llama 3.1 405B). FP8 or NVFP4 KV cache and weight quantization shrink it at serving time. For context window techniques that apply to all architectures, see context window management.

Why does MoE not help with memory?
Mixture-of-Experts gets its efficiency from sparse activation. Mixtral-8x7B has 46.7 billion total parameters, with 8 experts in every FFN layer, and each token is routed to 2 of them: about 12.9 billion parameters per token. The compute cost looks like a 12.9B dense model. The quality looks like a much larger one.
Memory does not take part in this bargain, for two reasons.
All the experts must be resident. The router can send the next token to any expert, so every expert’s weights have to sit in GPU memory (or be streamed in, at a large latency cost). Mixtral’s memory footprint for weights is set by its 46.7B total parameters: about 93 GB in BF16, not the 26 GB that 12.9B active parameters would suggest.
The KV cache ignores routing. Routing selects which FFN blocks process a token. The attention layers are shared: every token passes through all of them and writes keys and values at every layer. KV cache per token is
\[\text{bytes per token} = 2 \times \text{layers} \times \text{KV heads} \times \text{head dim} \times \text{bytes per value}\]and the number of experts appears nowhere in it. Mixtral uses 32 layers, 8 KV heads (grouped-query attention) and a head dimension of 128, the same attention configuration as Mistral 7B. In BF16 that is 2 × 32 × 8 × 128 × 2 = 131,072 bytes, or 128 KiB per token. A full 32K-token context (Mixtral’s trained window) costs 4 GiB per sequence.
One long request does not overwhelm a 93 GB model. Concurrency does. The cache matches the weights at about 712,000 cached tokens, which is roughly 22 sequences at the full 32K window. A busy endpoint can easily have that many tokens in flight, which is why the cache, not the weights, decides how many requests fit.
graph TD
subgraph "What MoE sparsifies"
A[FFN compute<br/>8 experts, 2 active<br/>75% fewer FFN FLOPs]
end
subgraph "What it does not"
W[Expert weights<br/>all 8 resident<br/>about 93 GB BF16]
B[Attention KV cache<br/>128 KiB per token<br/>independent of routing]
end
W --> D[Memory left for KV<br/>caps batch size]
B --> D
D --> E[Small per-expert batches<br/>lower MoE efficiency]
Why the KV budget hurts MoE more than dense models
Take two 80 GB GPUs (160 GB) serving Mixtral in BF16. After 93 GB of weights and some headroom for activations, roughly 60 GB is left for cache. At 128 KiB per token that is about 458,000 tokens: around 14 requests at the full 32K window, or about 220 conversations of 2K tokens each.
That batch limit costs an MoE more than a dense model. During decoding, each expert only sees the tokens routed to it, roughly batch × top-k / experts. With top-2 of 8, a batch of 64 sequences gives each expert about 16 tokens per step, too few to keep its matrix multiplies efficient. Dense models reach good hardware utilization at smaller batches because every token hits the same weights. So the MoE’s compute advantage only shows up when the batch is large, and the batch is capped by whatever memory the weights leave for the KV cache.
That is the memory wall: sparse compute, dense memory, and a throughput ceiling set by the cache budget.
Four MoE and dense reference points, all at 16-bit precision:
| Model | Total params | Active params | Attention | KV cache per token |
|---|---|---|---|---|
| Mixtral-8x7B | 46.7B | 12.9B | GQA, 32 layers, 8 KV heads | 128 KiB (0.131 MB) |
| Grok-1 | 314B | about 25% of weights | GQA, 64 layers, 8 KV heads | 256 KiB (0.262 MB) |
| DeepSeek-V3 | 671B | 37B | MLA, 61 layers | about 70 KB |
| Llama 3.1 405B (dense) | 405B | 405B | GQA, 126 layers, 8 KV heads | about 516 KB |
Grok-1’s figure comes from its released configuration (64 layers, 8 KV heads, head dimension 128); xAI describes 25% of weights as active per token. DeepSeek-V3’s number is far lower because it uses MLA, a different attention design covered below.
What standard KV cache techniques exist, and what they leave
Three widely deployed techniques help. None of them changes the fact that MoE weights crowd out cache.
PagedAttention (vLLM) breaks the KV cache into fixed-size blocks that can sit anywhere in GPU memory, like virtual memory pages. The vLLM authors measured 60-80% of KV memory wasted by fragmentation and over-reservation in earlier systems, and cut that to under 4%, which gave 2-4x throughput at the same latency. PagedAttention does not reduce the size of the cache; it stops you wasting the memory you have.
Grouped Query Attention (GQA) shares key-value heads across groups of query heads. Llama 2 70B uses 8 KV heads for 64 query heads, an 8x reduction against full multi-head attention. Mixtral already uses GQA, which is why its cache is 128 KiB per token rather than 512 KiB. GQA is an architecture decision: converting an existing multi-head model needs additional training (the GQA paper “uptrains” checkpoints with about 5% of the original pretraining compute).
Multi-Query Attention (MQA) is the extreme case: all query heads share one KV head. Maximum savings, with a possible quality cost, and the same training-time constraint.
Multi-GPU layout matters for MoE. Large MoE models are spread over many GPUs, and the attention layout decides how the cache is stored. With tensor-parallel attention, each GPU holds a slice of the KV heads; once the tensor-parallel degree exceeds the number of KV heads (8 for Mixtral), engines such as vLLM replicate KV heads across GPUs and the duplicate copies eat cache memory. A common alternative for large MoE models runs attention data-parallel (each GPU holds the full cache for its own requests) and the experts expert-parallel, with an all-to-all exchange to send tokens to their experts. That avoids replication but makes balancing cache load across attention ranks your problem.
What PiKV adds
PiKV (open source at github.com/NoakLiu/PiKV) is a research library aimed at KV cache management for MoE serving across GPUs and nodes. Its authors argue that MoE sparsifies computation while the KV cache stays dense and globally synchronized, and they combine four parts:
- Expert-sharded KV storage that partitions the cache across GPUs.
- PiKV routing, which reduces how much KV each token has to access.
- PiKV scheduling, which adaptively keeps the entries most relevant to current queries and evicts the rest.
- Compression modules, including low-rank (LoRA- and SVD-style) and quantization options.
The project README reports up to 2.2x faster inference with 65% lower memory for its quantized variant, against a standard MoE baseline. Treat these as the authors’ own numbers on their setup: the paper describes PiKV as still under development, and eviction is lossy by design, so measure quality on your own tasks before relying on it.
What is DeepSeek’s MLA, and how much does it save?
Multi-Head Latent Attention (MLA) takes a different approach: instead of managing full KV tensors better, it changes what gets cached.
Standard multi-head attention caches K and V at their full size, KV heads × head dimension per layer. For a model with 32 heads of dimension 128 (4,096 total) and no GQA, that is 4,096 × 2 (K and V) × 2 bytes = 16 KB per token per layer, and it multiplies across every layer.
MLA projects each token’s keys and values through a learned low-rank matrix and caches only the compressed latent. DeepSeek-V3 has 128 attention heads of dimension 128, but per layer it caches a 512-dimensional latent plus a 64-dimensional key used for rotary position embeddings. At attention time the latent is projected back up, or the up-projection is folded into the query and output projections. The projection matrices are trained with the rest of the model, so the model learns what to keep.
The result for DeepSeek-V3: (512 + 64) × 61 layers × 2 bytes = about 70 KB per token, versus about 516 KB for Llama 3.1 405B. That is 7.3x smaller (an 86% reduction). For one 128K-token sequence (131,072 tokens), that is about 9 GB of cache versus about 68 GB. The often-quoted 93.3% figure comes from the DeepSeek-V2 paper and compares DeepSeek-V2 with DeepSeek’s own dense 67B model.
The trade-offs: MLA does not work directly with standard rotary position embeddings (RoPE), so DeepSeek adds a separate “decoupled” RoPE key, which is the extra 64 dimensions above. And MLA is normally chosen at pretraining time. Conversion methods exist, such as TransMLA (GQA to MLA) and MHA2MLA, but they need fine-tuning to recover quality.
What should practitioners deploying MoE models do?
Size the cache budget first. Compute weights at your chosen precision, subtract from total GPU memory with headroom for activations, and divide by KV bytes per token. That number, not the active parameter count, tells you how many concurrent tokens a node can serve.
If you are serving Mixtral or a similar standard-attention MoE model:
- Serve with vLLM or SGLang, which use paged KV memory by default.
- Quantize the KV cache. FP8 halves it against BF16. On Blackwell GPUs, NVIDIA reports that NVFP4 KV cache cuts it by up to 50% again versus FP8. Check accuracy on your tasks.
- Quantize the weights to free room for cache. FP8 weights save about 47 GB on Mixtral, enough for roughly 356,000 more tokens of BF16 cache. The quantization decision tree covers which method fits which hardware.
- Avoid tensor-parallel degrees above the KV head count, or switch to data-parallel attention with expert-parallel FFNs.
- Treat expert-aware sharding and eviction (PiKV and similar) as experiments: benchmark throughput and quality before adopting them.
If you are choosing a model to deploy:
- MLA models such as DeepSeek-V3 carry a much smaller cache per token, which translates directly into larger batches per GPU.
- Standard GQA models such as Mixtral have mature tooling everywhere but need serving-side cache reduction to batch well.
If you are training a new MoE model:
- Decide on MLA or aggressive GQA at architecture selection time. Retrofitting is possible but costs extra training, and the serving savings accrue over the model’s whole lifetime.
Key takeaways
- MoE sparsifies compute, not memory. All experts stay resident, and the KV cache depends only on the attention configuration. Mixtral caches exactly as much per token as Mistral 7B.
- The cache caps the batch, and the batch decides MoE efficiency. Each expert sees only about top-k / experts of the batch, so MoE models need large batches, and large batches need KV memory that the weights have already claimed.
- Paging is necessary but does not shrink anything. PagedAttention removes waste; GQA, MQA and MLA reduce size; KV and weight quantization buy room at serving time.
- MLA is the architectural fix. DeepSeek-V3’s cache is about 7.3x smaller per token than Llama 3.1 405B’s.
- PiKV is promising research, not yet a default. Self-reported gains, lossy eviction, active development.
Further reading
- Context window management: general KV cache optimization for agents
- TurboQuant: KV cache compression: quantizing the cache itself
- NanoQuant: sub-1-bit quantization: another approach to memory reduction through extreme quantization