11 minute read

Weight quantization gets all the attention. Quantize to INT8, maybe INT4, watch the benchmark score. But model weights are a one-time cost. The KV cache grows with every token, and that’s where the real pressure lives at scale.

A Llama 3.1 70B request at 128K context consumes roughly 43 GB of BF16 KV cache alone. Serve four concurrent requests at that length and you need about 170 GB just for the cache, more than the ~141 GB of BF16 model weights. Weight quantization doesn’t touch that number. TurboQuant does.

TL;DR: TurboQuant (Google, ICLR 2026) compresses the KV cache to about 3 bits per value in two stages: a random rotation plus PolarQuant for the main compression, then a 1-bit QJL correction of the residual error. Storing 3 bits instead of 16 is roughly a 5x cut (Google reports at least 6x on its needle-in-a-haystack tests), with no measured accuracy loss on Google’s long-context benchmarks. Google’s “up to 8x faster” figure is for computing attention logits with 4-bit keys on H100, against 32-bit keys. vLLM merged a TurboQuant backend in April 2026. LookaheadKV (arXiv:2603.10899, also ICLR 2026) takes a complementary route, cutting the cost of deciding which KV entries to evict by up to 14.5x. The two techniques solve different problems and can be stacked.

A cross-section of layered data storage media under laboratory lighting, most layers compressed to ultra-thin densely packed sheets while standard ...


Why KV cache compression is different from weight quantization

Weight quantization doesn’t solve your long-context problem. On Llama 3.1 70B the KV cache costs about 0.33 MB per token in BF16 (2 × 80 layers × 8 KV heads × 128 head dimension × 2 bytes). At 8K context that is about 2.7 GB per request, or about 86 GB for a batch of 32, which is more than an 80 GB GPU holds before you load a single weight. In long-context serving the cache often takes most of the GPU memory left after weights, and it grows linearly with every token.

The structural difference is what matters. Model weights are static: quantize once, done. The KV cache is dynamic. It grows with sequence length, fluctuates with batch size, and gets written and read on every forward pass. That changes the requirements: you need low-overhead encoding and decoding at inference time, not a one-shot calibration pass at deployment.

There’s also a targeting problem with weight-only quantization. It helps when your batch is small and your sequences are short. Once you push toward 32K+ context or large batches on long documents, the cache is where you’re losing. The concrete comparison: INT8 weight quantization on a 70B model saves about 70 GB, once (141 GB down to 71 GB). 3-bit KV compression saves about 35 GB per 128K-context request (43 GB down to 8 GB), and that saving multiplies with every concurrent request.


What TurboQuant actually does

TurboQuant gets the KV cache to about 3 bits per value with no measured accuracy loss on Google’s benchmarks by solving a problem most 3-bit approaches get wrong. Standard quantizers introduce systematic bias into attention score computation. TurboQuant is designed not to. The method is described in arXiv:2504.19874 and Google’s research blog post.

The algorithm runs in two stages, after a random rotation of each vector that spreads outliers and makes the coordinates’ distribution predictable.

PolarQuant is the main compression pass and uses most of the bit budget. Instead of quantizing KV vectors in Cartesian space (where outliers blow up quantization error), PolarQuant converts pairs of coordinates into polar form, recursively. After the rotation, the angles follow a concentrated, predictable distribution. Because the angles cluster, PolarQuant can drop the per-block normalization constants that conventional quantizers store in full precision, overhead Google puts at 1 to 2 extra bits per number. You end up close to genuine 3-bit storage, not “3-bit plus scale factors buried in the overhead.”

QJL (quantized Johnson-Lindenstrauss) is a 1-bit cleanup pass. Any quantizer leaves residual error. QJL projects that residual through a random matrix and stores only the sign of each projection, 1 bit per value, with an estimator that keeps attention scores unbiased. Unbiased means the errors do not push attention scores systematically up or down. They still add variance, which is why the benchmark results matter as much as the math.

Google reports up to 8x faster computation of attention logits on H100 for 4-bit TurboQuant against unquantized 32-bit keys. Treat that as a kernel-level figure against a 32-bit baseline, not an end-to-end speedup over a BF16 serving stack. Decode on long sequences is bound by memory bandwidth rather than FLOPs, so reading fewer bytes per token does help throughput. The method was evaluated on LongBench, Needle In A Haystack, ZeroSCROLLS, RULER and L-Eval using open models (Gemma, Mistral and Llama-3.1-8B), including needle retrieval tasks designed to break approximate attention.

One thing practitioners often miss: no retraining, no calibration dataset, no fine-tuning. You still need attention kernels that read the compressed format, which is what the vLLM backend provides.

Memory breakdown: Llama 3.1 70B, 128K context, one H100 (80 GB)

Without KV compression:
  Model weights (4-bit AWQ):    ~38 GB
  KV cache per request (BF16):  ~43 GB
  Activations + misc:           ~5 GB
  ────────────────────────────────
  One request:                  ~86 GB   <- does not fit

With 3-bit KV (16/3 ≈ 5.3x smaller):
  Model weights (4-bit AWQ):    ~38 GB
  KV cache per request:         ~8 GB
  Activations + misc:           ~5 GB
  ────────────────────────────────
  Room for KV after weights:    ~37 GB   -> about 4 concurrent 128K requests

Effect: a single H100 goes from not fitting one 128K request
to fitting about four.

LookaheadKV: predicting what to keep vs. compressing what you keep

TurboQuant compresses everything in the cache. LookaheadKV asks a prior question: which entries deserve to stay in the cache at all?

KV cache eviction (dropping low-importance entries to cap cache size) has a longstanding accuracy problem. Eviction decisions have to be made during the prefill phase, before the model generates any response. But importance scores depend on what queries will be issued during generation, and those don’t exist yet. Existing heuristics (recency weighting, attention sink patterns) work adequately at short context but degrade as sequences lengthen.

LookaheadKV trains lightweight, parameter-efficient modules that predict future query importance without generating draft tokens. The modules augment each transformer layer and learn to approximate true importance scores. The design choice is what the authors call a “surrogate future response”: the modules simulate what the model would ask of the cache, without actually running generation. The paper reports eviction cost up to 14.5x lower than prior costly approximation methods, with better accuracy than cheap heuristics.

The practical payoff is faster time-to-first-token at long contexts. The prefill phase can confidently drop entries that would never be attended to, rather than holding everything just in case. This is different from what TurboQuant provides. TurboQuant doesn’t reduce cache entry count; it reduces per-entry memory cost. Both appeared at ICLR 2026, and they’re solving orthogonal constraints.


A decision matrix: which compression technique for which constraint

The choice between eviction and quantization maps onto deployment constraints fairly cleanly:

flowchart TD
    A[KV cache is your bottleneck] --> B{What's the primary constraint?}
    B --> C[Memory capacity\ne.g. won't fit in VRAM]
    B --> D[Latency / TTFT\ne.g. too slow to respond]
    B --> E[Both: long context\n+ high throughput]

    C --> F{Acceptable accuracy\ntrade-off?}
    F --> G[Near-zero loss required] --> H[TurboQuant ~3-bit\n~5x memory reduction]
    F --> I[Want lowest risk] --> J[FP8 KV in vLLM\n2x, long-standing default]

    D --> K{Is eviction already\nin your stack?}
    K --> L[No] --> M[LookaheadKV\nup to 14.5x cheaper eviction]
    K --> N[Yes, but inaccurate] --> M

    E --> O[LookaheadKV eviction\n+ TurboQuant on kept entries\nmaximum compression]

    H --> P[Validate on your task:\nneedle retrieval, multi-hop]
    J --> P
    M --> P
    O --> P

TurboQuant is the right choice when sequences are long, you need to maximize concurrent requests in VRAM, and dropping any context is not acceptable (RAG over full documents, legal document review, long multi-turn sessions). All tokens stay; each costs about a fifth of the memory.

LookaheadKV fits when you’re optimizing TTFT on a fixed context budget and your workload has natural redundancy, such as long system prompts or retrieved chunks that are mostly irrelevant. Cheaper eviction decisions shrink the prefill overhead, which is where the TTFT gain comes from. You do give up some context, so accuracy on tasks that need every token will degrade.

Combining them makes sense for high-throughput long-context serving where both density and speed matter. Evict the low-importance entries first, then quantize what remains. The gains compound.

Using neither is also a valid call. FP8 KV cache (--kv-cache-dtype fp8 in vLLM) halves KV memory relative to BF16 and is the long-standing low-risk option. TurboQuant has been in vLLM mainline since April 2026 as its own attention backend, so choosing it is now a question of validation effort rather than running a fork. For how KV compression stacks with weight formats, see the quantization decision tree.

One constraint that overrides everything else: if your application needs to retrieve specific facts buried deep in long context (the needle-in-a-haystack pattern), test your compression method explicitly on that task. Aggressive eviction schemes can silently discard the needle. TurboQuant’s unbiased estimator is designed to prevent this, but verify before deploying.


FAQ

Does TurboQuant require retraining or fine-tuning the model? No. TurboQuant operates entirely at inference time. You quantize the KV cache on the fly with no changes to model weights, no calibration dataset, no fine-tuning pass.

Can TurboQuant and LookaheadKV be combined? Yes. LookaheadKV drops low-importance entries after prefill; TurboQuant compresses the entries that remain. They target different parts of the memory problem and have no architectural conflict.

What context lengths justify KV cache compression over weight quantization? Compare total cached tokens against weight memory. On Llama 3.1 70B, at about 0.33 MB per token in BF16, the cache passes the BF16 weights (~141 GB) at roughly 430K cached tokens and INT8 weights (~71 GB) at roughly 215K, summed across all concurrent requests. One 128K request needs about 43 GB of cache, which weight quantization does nothing to reduce.

Is 3-bit KV quantization safe for all tasks? Google evaluated TurboQuant on LongBench, Needle In A Haystack, ZeroSCROLLS, RULER and L-Eval with Gemma, Mistral and Llama-3.1-8B and reports no measured accuracy loss. Tasks with high attention sensitivity to specific tokens, such as precise fact retrieval and multi-hop reasoning, are the ones to stress-test before going to production.

Is TurboQuant in vLLM yet? Yes. vLLM merged a TurboQuant attention backend in April 2026 (PR #38479). You select it with --kv-cache-dtype, using presets that range from turboquant_k8v4 (FP8 keys, 4-bit values, about 2.6x compression) to turboquant_3bit_nc (3-bit keys and values, about 4.9x). The first release covered full-attention and sliding-window models, not hybrid attention-plus-Mamba models. Community implementations such as 0xSero/turboquant predate it.


Related reading: