Speculative decoding meets 4-bit quantization: the gains stack, but don’t multiply
“Speculative decoding used to be a research paper. Now it is a checkbox in vLLM.”
TL;DR
Speculative decoding and 4-bit weight quantization attack the same bottleneck. At small batch sizes every decode step streams all the weights from memory; quantization makes each read cheaper, and speculative decoding verifies several tokens per read. Because they target the same cost, the gains stack but don’t multiply. A 2025 study found EAGLE-2 adds less speedup on 4-bit models than on higher-precision ones, and its hierarchical fix reaches 2.78x over 4-bit autoregressive decoding on Llama-3-70B. ML-SpecQD uses a 4-bit copy of the model as a training-free draft (up to 2.72x over BF16 on a client CPU). P-EAGLE drafts all tokens in one pass for another 1.1-1.4x over EAGLE-3. For the foundational mechanics, see speculative decoding fundamentals.

What changed since the fundamentals post?
Four pieces of work clarified how the two techniques interact and made the combination easier to deploy.
ML-SpecQD (arXiv 2503.13565, Intel researchers, March 2025) uses quantization to build the draft rather than to shrink the target. It direct-casts the BF16 target’s weights to MXFP4 and uses that 4-bit copy as the draft, so no draft model has to be trained. It then speeds up the draft with a second, smaller draft (also MXFP4), which makes the speculation multi-level. The target stays BF16, so outputs match BF16 greedy decoding exactly.
Speculative decoding meets quantization (arXiv 2505.22179, May 2025) measured what happens when you run EAGLE-2 on quantized Llama-3 targets. The finding: integration improves overall speedup, but EAGLE-2 adds less on top of 4-bit weight quantization than on top of 8-bit or 16-bit. The authors traced this to the cost of verifying tree-shaped drafts and proposed a hierarchical fix, HierSpec.
EAGLE-3 in vLLM. vLLM runs EAGLE-3 drafters through its speculative-decoding config, and the Speculators library (v0.3.0, December 2025) added end-to-end training of EAGLE-3 drafters that load straight into vllm serve. Pre-trained drafters exist for several popular models. This is what moved the technique from “interesting” to “adopted”.
P-EAGLE (arXiv 2602.01469, February 2026, from Amazon and NVIDIA researchers) attacks the draft model’s own latency. Standard EAGLE drafts tokens one at a time; P-EAGLE’s 4-layer drafter generates all K draft tokens in a single forward pass. It has been in vLLM since v0.16.0.
Why the gains don’t multiply
At small batch sizes, a decode step is limited by memory bandwidth: the GPU must stream every weight to produce one token per sequence. Take Llama-3-70B on a single A100. In FP16 the weights (about 140 GB) don’t even fit. With 4-bit weights (W4A16, about 35 GB) they do, and at roughly 2 TB/s of bandwidth, streaming them takes about 17 ms, a ceiling of about 58 tokens/s at batch 1.
Both techniques attack that same 17 ms:
- Quantization cuts the bytes read per step, roughly 4x against FP16.
- Speculative decoding verifies several draft tokens in one step, so each weight read produces more than one token. While the step is bandwidth-bound, checking 6 tokens costs about the same as producing 1.
The catch is the second condition. A 4-bit step moves fewer bytes but does extra dequantization and matrix math per byte, so it becomes compute-bound with fewer tokens in flight. EAGLE-2 verifies a whole tree of candidates per step (24 to 60 tokens in the paper’s configurations), and on a 4-bit target that verification is no longer close to free. The paper measured EAGLE-2’s verification time at 1.53x that of vanilla sequence speculation on the W4A16 70B model, while vanilla speculation spent 1.93x longer drafting. On this target, plain sequence speculation (with Llama-3-8B as the draft) actually beat EAGLE-2.
HierSpec combines the two: EAGLE-2 drafts a tree for a W4A16 Llama-3-8B, the 8B verifies the tree and hands a short sequence to the 70B, and the 70B only verifies a sequence. Results on the W4A16 Llama-3-70B, averaged across tasks on one A100:
| Method (W4A16 Llama-3-70B target) | Decoding speed | Speedup over 4-bit autoregressive |
|---|---|---|
| Autoregressive decoding | about 35 tokens/s | 1.0x |
| EAGLE-2 (best configuration) | about 75 tokens/s | about 2.1x |
| Vanilla speculative decoding, 8B draft (best) | about 81 tokens/s | about 2.3x |
| HierSpec | 98.2 tokens/s | 2.78x |
The 98.2 tokens/s and the 1.31x and 1.21x margins over the best EAGLE-2 and vanilla configurations are from the paper; the other rows are derived from those ratios.
There is a second interaction worth separating out. Quantization also frees memory for larger batches, which raises throughput. But larger batches push the GPU toward compute-bound, where speculative decoding helps less: the EAGLE-3 paper reports speedups up to 6.5x at small batch sizes but 1.38x throughput at batch size 64 in SGLang. Decide what the deployment is for. For latency at low concurrency, stack speculative decoding on a 4-bit target. For throughput at high concurrency, quantization plus larger batches does most of the work.
ML-SpecQD: quantization as the draft
ML-SpecQD’s results come from a client CPU (an Intel Core Ultra 9 285K), with a BF16 Qwen2.5-Coder 7B target on a code generation benchmark:
| Configuration | Geomean speedup over BF16 greedy | Draft acceptance |
|---|---|---|
| Speculative decoding with Qwen2.5-Coder 0.5B draft | 2.64x | about 62% |
| Speculative decoding with MXFP4 copy of the 7B as draft | 2.03x | about 91% |
| Multi-level: MXFP4 7B draft, itself drafted by MXFP4 0.5B | 2.72x |
Two lessons. First, a 4-bit copy of the target is a very well-aligned, training-free draft, which matters for model families with no small sibling (the paper notes Code Llama’s smallest model is 7B). Second, when a good small draft model already exists, the multi-level combination only adds a little (2.72x versus 2.64x). On the paper’s Llama2 7B question-answering setup, which used a 68M draft, the separate draft gave a 1.81x geomean speedup.
P-EAGLE: removing the sequential draft
EAGLE-style drafters generate draft tokens one after another, so deeper speculation means a longer drafting phase. P-EAGLE trains the drafter to predict up to K tokens in parallel in one forward pass. The paper reports 1.10-1.36x over autoregressive EAGLE-3 across GPT-OSS 120B, GPT-OSS 20B and Qwen3-Coder 30B. The vLLM blog reports 1.55-1.69x on GPT-OSS 20B at concurrency 1 on a B200, falling to 1.05-1.25x at concurrency 64.
Summary of reported results
| Configuration | Speedup | Baseline | Hardware | Source |
|---|---|---|---|---|
| ML-SpecQD multi-level (MXFP4 drafts, BF16 target) | up to 2.72x | BF16 greedy decoding | Intel client CPU | arXiv 2503.13565 |
| EAGLE-2 on 4-bit Llama-3-70B | about 2.1x | 4-bit autoregressive | A100 | arXiv 2505.22179 |
| HierSpec on 4-bit Llama-3-70B | 2.78x | 4-bit autoregressive | A100 | arXiv 2505.22179 |
| EAGLE-3 | up to 6.5x (batch 1); 1.38x throughput at batch 64 | No speculation | GPU | arXiv 2503.01840 |
| P-EAGLE | 1.10-1.36x (paper); 1.05-1.69x (vLLM blog) | EAGLE-3 | B200 for the blog numbers | arXiv 2602.01469 |
These rows use different models, hardware and baselines, so they cannot be multiplied together into one end-to-end figure.
How should you enable this in production?
Step 1: Enable an EAGLE-3 drafter in vLLM. This is the lowest-effort change for latency-sensitive endpoints. Use a published drafter for your model if one exists, or train one with Speculators.
Step 2: Combine with 4-bit weights where memory or cost demands it. GPTQ, AWQ and similar 4-bit formats run natively in vLLM; the quantization decision tree covers which to pick for your hardware. Expect the speculative gain on a 4-bit target to be smaller than on FP16, and tune draft depth and tree size again, since smaller trees suit 4-bit targets better. If your model has no drafter, a 4-bit copy of the target is a training-free option.
Step 3: Profile acceptance rates and concurrency. The benefit varies by workload: code and structured output accept more draft tokens than open-ended text, and every technique here loses ground as concurrency rises. Measure on your own traffic at your real batch sizes.
Step 4: Evaluate the quantized target, not the drafter. With standard (lossless) verification, speculative decoding reproduces the target model’s output distribution exactly, so it adds no quality risk of its own. Quantizing the target does change outputs; run your quality evaluations on the 4-bit target before enabling the combination.
Key takeaways
- The gains stack but don’t multiply. Both techniques reduce the cost of streaming weights at small batch sizes, so 4-bit weights leave less for speculative decoding to amortize.
- Draft shape matters on quantized targets. Large tree drafts make verification compute-bound on 4-bit models; sequence drafts or a hierarchical design (2.78x on 4-bit Llama-3-70B) recover the gap.
- Quantization can build the draft. A 4-bit copy of the target is a training-free, well-aligned draft (ML-SpecQD, up to 2.72x over BF16 on a client CPU).
- P-EAGLE removes the sequential draft. 1.1-1.4x over EAGLE-3 in the paper, up to 1.69x at low concurrency in vLLM.
- Profile your workload. Low concurrency and predictable outputs benefit most.
Further reading
- Speculative decoding fundamentals: the mechanics this post builds on
- QuantSpec: self-speculative decoding with a quantized KV cache
- SDSL: speculative decoding scaling laws: how big the draft model should be
- NanoQuant: sub-1-bit quantization: extreme quantization approaches
- Token efficiency optimization: reducing cost through smarter token usage