The residual stream revelation: why KV cache may be theoretically unnecessary
Every team running LLM inference at scale has the same conversation. Someone opens a memory profiler, sees the KV cache consuming most of the GPU memory budget, and asks: can we make this smaller? Then come the follow-up questions: quantize it, evict it, compress it, page it, offload it to CPU. The conversation is always about how to manage the cache, never about whether the cache stores something genuinely irreplaceable.
A March 2026 preprint, arXiv:2603.19664 (“The Residual Stream Is All You Need”), asks that second question directly: is the information in the KV cache actually new? Its answer is no. That part is correct and, on reflection, unsurprising. The paper’s memory claims are the interesting part, and they need a closer read.
TL;DR
The KV cache holds nothing the model cannot recompute. Keys and values at every layer are deterministic projections of the residual stream, so a model decoding without a cache produces the same tokens, only slower. The preprint confirms this on six small models and proposes KV-Direct, which checkpoints one residual vector per token instead of K/V pairs: 5 KB versus 136 KB per token on Gemma 3-4B, 27x smaller. Two caveats matter for production. Rebuilding K and V costs compute that grows with how much you recompute, and the paper does not fully explain how one vector per token yields K and V at every layer. Treat KV-Direct as a memory-for-compute trade to benchmark, not as proof that the cache is obsolete. For memory management at the application layer, see context window management.

What is the residual stream, and why does every layer write to it?
The residual stream is the running state vector that flows through every transformer layer. When you feed a sequence of tokens to a transformer, each token starts as an embedding. From there, every layer reads from that vector, computes something, and adds its result back. The output of layer N becomes the input to layer N+1.
This “add and pass” design is a residual connection, the same idea ResNets introduced for vision models. Each layer contributes its delta to a shared stream rather than transforming the entire state from scratch.
Token embeddings
│
▼
┌─────────────────────────────────────────┐
│ Layer 1 │
│ ┌──────────────┐ ┌───────────────┐ │
│ │ Self-Attention│ │ FFN │ │
│ └──────┬───────┘ └───────┬───────┘ │
│ │ (+= residual) │ │
└─────────┼───────────────────┼──────────┘
│ + │
└────────┬──────────┘
│ ← residual stream: complete state at this position
▼
┌─────────────────────────────────────────┐
│ Layer 2 reads the same stream │
│ ... │
└─────────────────────────────────────────┘
A 2021 paper from Anthropic researchers (Elhage, Nanda, Olsson et al., “A Mathematical Framework for Transformer Circuits”) formalized this: attention heads read from the residual stream, perform their computations, and write back. Nothing is created outside the stream. Everything goes through it.
The implication is that at any given layer and position, the residual stream is the complete representation of what the model has computed for that token so far. Not a summary. The entire latent state.
How are K and V derived from the residual stream?
When an attention layer computes attention, it projects the residual stream through learned weight matrices:
K_l = RotaryEmbed(LayerNorm(r) @ W_K)
V_l = LayerNorm(r) @ W_V
Where r is the residual vector at that token position and layer, and W_K, W_V are weight matrices frozen after training: they never change during inference.
Given r, you can compute K_l and V_l exactly. The operations are deterministic: layer normalization, then a matrix multiply, then positional rotations for keys. No stochasticity, no learned dynamics, no hidden state. The same r always produces the same K and V.
The KV cache stores K_l and V_l for every layer and every past token. But since those values are deterministic projections of r, and r flows through the forward pass anyway, the cache is storing derived quantities that can always be reconstructed. That’s what arXiv:2603.19664 calls theoretical redundancy, not a claim that caching is wasteful in practice, but a claim that the cache contains no information the residual stream doesn’t already carry.
What does “theoretically redundant” actually mean?
The argument is about determinism. At layer l, a token’s key and value are fixed functions of that token’s residual at layer l. That residual is in turn a fixed function of the input tokens and the weights. So the cache holds derived data: given the tokens and the weights, you can always rebuild it exactly.
The authors test this in two ways. Cross-task residual patching at every layer gives a KL divergence of exactly zero between the patched and original output distributions. Then they drop the cache and recompute everything from scratch at each decoding step, generating 30 greedy tokens on six models from four families (SmolLM2, Qwen2.5 and Qwen3, a DeepSeek-R1 distill of Qwen, and Gemma 3), from 135M to 4B parameters.
Token identity: 30 of 30 on every model. The from-scratch path ran 1.7 to 3.8x slower than cached decoding.
That second result is what you would predict. Decoding without a cache is how a transformer behaves before you add the optimization, and the cache exists to avoid exactly this recompute. The result confirms the redundancy cleanly, but it does not by itself show a cheaper path. One inconsistency is worth knowing about: in a separate evaluation the paper reports Gemma 3’s cache-free path dropping HellaSwag accuracy from 49.2% to 25.0%, and it does not reconcile that with the token-identity result.
What KV-Direct stores, and what the 27x leaves out
KV-Direct is the paper’s practical proposal: checkpoint residual vectors instead of K/V pairs, and recompute K and V on demand. The per-token arithmetic for Gemma 3-4B (34 layers, 4 KV heads, head dimension 256, hidden size 2,560, bf16):
Standard KV cache per token: 2 × 34 layers × 4 KV heads × 256 × 2 bytes = 139,264 bytes ≈ 136 KB
One residual vector per token: 2,560 × 2 bytes = 5,120 bytes ≈ 5 KB
Ratio: ≈ 27x
The paper reports peak memory of 42 MB over a 20-turn conversation, against more than 103 MB for a standard cache, and a 100% token match where five eviction baselines (H2O, StreamingLLM, SnapKV, TOVA, window-only) fall to 5 to 28%.
Two things to check before you plan around the 27x.
One vector for all layers is asserted, not derived. The paper states that “a single residual vector serves all L layers.” But K and V at layer l come from the residual at layer l, and the upper-layer residuals depend on attention over other tokens. Getting them from one stored vector means running the layers again, which is a forward pass, not a lookup. If you store a residual per layer instead, the cost on Gemma 3-4B is 34 × 5 KB ≈ 170 KB per token, more than the 136 KB K/V cache, because grouped-query attention already makes K and V smaller than the residual.
The speed results come from a laptop. All experiments ran on an Apple M3 Max with MLX. There, recomputing K/V for 500 evicted tokens took 0.17x the time of reading the cached tensors on the largest model, while for a single token recompute was slightly slower than the read. The paper does not test whether that crossover holds on a datacenter GPU, where HBM bandwidth and batch composition are very different.
Where recompute-instead-of-store could pay
The practical idea is not “stop using KV cache.” It is that storing and recomputing are both options, and the cheaper one depends on hardware and workload.
Long-context applications. For Qwen2.5-14B-Instruct-1M (48 layers, 8 KV heads, head dimension 128), the bf16 KV cache is about 192 KB per token, so a 1M-token context needs roughly 200 GB for the cache alone. A 10 KB residual per token would be about 10 GB. The catch is that rebuilding K and V over a long prefix is itself a large prefill, so the realistic design is hybrid: keep recent K/V in memory and checkpoint residuals for older context.
Memory-constrained deployments. Smaller per-token storage changes what fits in unified memory on edge devices and smaller GPU tiers, which is the kind of hardware the paper tested on. Long chat histories are the obvious use.
Latency-tolerant batch workloads. For interactive applications, a cache read is the safe default. For batch processing, document analysis or offline agent pipelines, extra compute per step is often acceptable if it buys memory headroom.
flowchart LR
A[Inference workload] --> B{Latency budget?}
B -->|Interactive, sub-200ms| C[Standard KV cache]
B -->|Batch or offline| D{Memory constrained?}
C --> E[Optimize: quantize, evict, page]
D -->|GPU memory abundant| C
D -->|Memory constrained| F[KV-Direct: checkpoint residuals]
F --> G[Recompute K and V on demand]
E --> H[TurboQuant / LookaheadKV / paged attention]
Which KV cache optimizations still matter?
All of them. The redundancy result does not change how they work.
For latency-sensitive inference, the cache’s speed advantage remains. The March 2026 survey arXiv:2603.20397 organizes KV cache optimization into five directions: cache eviction, compression, hybrid memory, novel attention mechanisms, and combination strategies. Each remains relevant wherever latency matters.
It is tempting to read the redundancy result as a guarantee for lossy compression. It is not one. When you quantize or evict cache entries, the model attends over the degraded cache, and that loss is real unless you pay to recompute from the residuals. What the result gives you is a fallback: an exact cache can always be rebuilt.
TurboQuant (Google, ICLR 2026) quantizes the KV cache to around 3 bits with no measured accuracy loss on Google’s long-context benchmarks, and Google reports at least 6x lower KV memory. That result comes from its quantizer design (random rotation, PolarQuant, and a 1-bit QJL correction of the residual error), not from the residual stream. The TurboQuant post covers it in detail.
LookaheadKV (arXiv:2603.10899, ICLR 2026) predicts which cache entries future tokens will need without running generation, cutting eviction cost by up to 14.5x. This remains valuable when the cache is retained and the question is which K/V pairs to keep. Under KV-Direct, the equivalent question becomes which residual checkpoints to retain, a simpler problem since residuals are uniform in size.
Directions this points to:
- Hybrid systems that keep recent K/V and checkpoint residuals for older context, with prefetch pipelines to hide recomputation cost
- Benchmarks on datacenter GPUs that measure the recompute-versus-read crossover at realistic batch sizes
- Edge LLM deployments where the latency trade is acceptable and the memory savings are decisive
The cache isn’t going away. It is an optimization that trades memory for speed, and it stores no information the model could not recompute.
For cost reduction approaches that complement this work at the application layer, see token efficiency optimization. For choosing a cache and weight quantization format by hardware, see the quantization decision tree.
FAQ
What is the residual stream in a transformer? The residual stream is the shared state vector that flows through every transformer layer. Each layer reads from it, computes a delta, and adds that delta back. At any point in the forward pass, the residual stream contains the complete representation of everything the model has processed. Elhage et al. (Anthropic, 2021) formalized this: all attention heads and FFN layers communicate exclusively through the residual stream.
Why is the KV cache theoretically redundant if it speeds up inference? Speed and information are separate properties. The KV cache contains no information the residual stream doesn’t already hold. K and V are deterministic projections of the residual at each layer, computable from frozen weight matrices. The cache trades memory for latency. Calling it “redundant” means it stores derived data, not that storing it is useless.
Can you run inference with no KV cache at all? Yes, and that has always been true: without a cache, the model recomputes keys and values for the whole prefix at every step. arXiv:2603.19664 (March 2026) confirms that decoding this way matched cached decoding for 30 greedy tokens on six models from 135M to 4B parameters, while running 1.7 to 3.8x slower.
What is KV-Direct? KV-Direct is the bounded-memory scheme proposed in arXiv:2603.19664. It stores one residual vector per token (5 KB on Gemma 3-4B) instead of all-layer K/V pairs (136 KB), a 27x reduction per token, and recomputes K and V on demand. The authors report 42 MB peak memory over a 20-turn conversation versus more than 103 MB with a standard cache. The paper does not fully explain how a single vector per token yields K and V at every layer, so check the method before relying on the numbers.
Does this mean KV cache compression can achieve zero information loss? No. Lossy KV compression loses information in the compressed cache, and the model attends over that compressed cache. The redundancy result only says you could rebuild an exact cache by recomputing, at a compute cost. TurboQuant’s near-zero measured accuracy loss comes from its quantizer design, not from the residual stream.