Speculative decoding in 2026: Saguaro, Nightjar, and the universal draft model
“In standard speculative decoding, the draft model sits idle while the target verifies. That idle time is the bottleneck Saguaro goes after.”
TL;DR
Three papers solve three different speculative decoding bottlenecks. Saguaro (ICLR 2026, Kumar, Dao and May) fills the draft model’s idle time during verification and reports 255.8 tok/s on Llama-3.1-70B at batch size 1, 4.7x over autoregressive decoding. Nightjar adapts speculation depth to server load, with up to 14.8% higher throughput and up to 20% lower latency than fixed-depth speculation. Intel and Weizmann’s SLEM algorithm lets a draft model accelerate a target with a different vocabulary, at up to 2.8x. Each targets a different deployment scenario, and understanding which one fits your serving stack is the point of this post. For the foundational inference techniques these build on, see 14 techniques for high-performance ML inference.

Why does speculative decoding matter more in 2026 than in 2024?
Because inference is now the dominant cost. Deloitte’s 2026 TMT Predictions expects inference to account for roughly two-thirds of all AI compute in 2026, up from about one-third in 2023, and the market for inference-optimized chips to exceed $50 billion in 2026.
Speculative decoding directly attacks this cost center. At low batch sizes, a 2x decode speedup lets the same GPUs serve about twice the latency-bound traffic. At batch size 1 (the latency-sensitive interactive case), standard speculative decoding already delivers 2-3x speedups. The three papers covered here push that envelope further: Saguaro to 4.7x, Nightjar to variable-load scenarios where fixed speculation hurts, and universal drafting to model pairs that were previously incompatible.
The production ecosystem has caught up too. EAGLE-3 (NeurIPS 2025) is supported in vLLM, SGLang and TensorRT-LLM. Amazon SageMaker AI picks EAGLE-2 or EAGLE-3 automatically based on model architecture. P-EAGLE, in vLLM since v0.16.0, adds parallel drafting, with vLLM reporting up to 1.69x over EAGLE-3 on B200. Speculative decoding moved from research novelty to production infrastructure between 2024 and 2026.
How does standard speculative decoding work?
A small draft model generates K candidate tokens. The large target model then verifies all K tokens in a single forward pass. Rejection sampling at each position ensures the output distribution is mathematically identical to sampling from the target alone. No quality loss.
Standard speculative decoding (one round):
Draft model (1B): [t1] → [t2] → [t3] → [t4] (fast, sequential)
|
Target model (70B): ══════ verify all 4 in parallel ═══ (one forward pass)
|
Result: [t1 ✓] [t2 ✓] [t3 ✗] → resample [t3*]
(if all 4 pass, the target adds a bonus token)
Why it works: at batch size 1 the target model’s forward pass costs nearly the same whether it processes 1 or K tokens, because it is memory-bandwidth bound. With K = 4 and a per-token acceptance rate between 0.6 and 0.8, the expected output is about 2.3 to 3.4 tokens per target call instead of 1.
Rough acceptance rates by task (illustrative; they vary widely by model pair, so measure on your own traffic):
| Task type | Acceptance rate (α) |
|---|---|
| Code, structured output | 0.75-0.85 |
| General chat, instructions | 0.60-0.80 |
| Creative writing, domain-specific | 0.50-0.65 |
At low acceptance rates, the cost of drafting can outweigh the savings; where the break-even sits depends on how cheap the draft is relative to the target. For how large the draft should be, see SDSL’s scaling laws for speculative decoding. The three papers below each address a specific failure mode of the basic approach.
What does Saguaro change by speculating on speculations?
Saguaro (arXiv:2603.03251, ICLR 2026) is the paper’s optimized algorithm for “speculative speculative decoding” (SSD). It removes the sequential dependency between drafting and verification. In standard speculative decoding, the draft model sits idle during the entire verification phase. Saguaro fills that dead time.
sequenceDiagram
participant D as Draft model (1B)
participant T as Target model (70B)
participant C as Speculation cache
Note over D,T: Standard SD: draft idles during verify
D->>T: Send draft tokens [t1..t4]
Note over D: Idle waiting...
T->>D: Verified: accept 3, bonus token t*
Note over D,T: Saguaro: draft works during verify
D->>T: Send draft tokens [t1..t4]
D->>C: Predict likely outcomes (k=2,t*=A), (k=3,t*=B)
D->>C: Generate continuations for each outcome
T->>C: Verified: accept 3, bonus=B
C->>T: Cache hit → instant next draft
The mechanics: while the target model verifies round T, the draft model predicts the most likely verification outcome as a (k, t) pair, where k is the number of accepted tokens and t is the bonus token. It generates continuations for several likely outcomes and stores them in a cache. When verification completes, a cache lookup returns the next draft with near-zero latency.
Three optimizations make this work:
- Geometric fan-out distributes the speculation budget across outcomes weighted by acceptance probability. The paper reports predicting the bonus token with up to 90% accuracy.
- Acceptance-cache tradeoff sampling downweights the draft’s top tokens, trading a little acceptance rate for a higher cache hit rate, with a tunable parameter.
- Adaptive fallback picks the backup speculator by batch size: a higher-quality fallback at small batches, a faster one above a critical batch size where overhead matters more.
Measured on Llama-3.1-70B Instruct (4x H100, tensor parallel) with Llama-3.2-1B as the draft on a separate H100, batch size 1, averaged over four datasets (paper appendix B.3):
| Method | Tokens/sec | Speedup vs AR |
|---|---|---|
| Autoregressive | 54.7 | 1.0x |
| Standard SD | 161.8 | 3.0x |
| Saguaro | 255.8 | 4.7x |
The abstract’s headline is more conservative: about 30% faster on average than the strongest speculative decoding baselines, and up to 5x faster than autoregressive decoding in open-source engines.
The catch: Saguaro requires a dedicated GPU for the draft model, separate from the target model’s hardware. Standard SD often collocates draft and target on the same device. This is a deployment architecture change, not just a configuration flag.
How does Nightjar adapt to changing server load?
Nightjar (arXiv:2512.22420) solves a problem Saguaro ignores: what happens when your server load changes. Standard speculative decoding with a fixed speculation length helps at low request rates but hurts at high ones. The target model becomes compute-saturated at large batch sizes, and speculation adds overhead without benefit.
Nightjar uses a multi-armed bandit (MAB) planner that continuously adjusts the speculation depth γ ∈ {0, 1, …, Γmax} based on current batch size. Setting γ=0 disables speculation entirely. The planner uses batch size as its primary context signal and balances exploration and exploitation to find a good depth for each load level. The thresholds in the diagram below are illustrative; the planner learns them online.
graph TD
A[Incoming requests] --> B{Current batch size?}
B -->|Small| C[High γ<br/>Full speculation<br/>Max latency benefit]
B -->|Medium| D[Medium γ<br/>Reduced speculation<br/>Balance throughput/latency]
B -->|Large| E[γ = 0-1<br/>Minimal or no speculation<br/>Avoid overhead]
E -->|Memory pressure| F[Offload draft model to CPU<br/>Reclaim GPU for KV cache]
F -->|Load drops| D
When the server is heavily loaded and speculation is disabled (γ=0), Nightjar does something unusual: under GPU memory pressure it offloads the draft model to CPU and hands that memory to the KV cache, which allows larger batches. The paper reports that expanding the KV cache this way takes 143.9 ms and shrinking it back takes 11.9 ms with a Triton kernel.
Results under dynamic load (Poisson arrivals plus a segment of the Azure LLM inference trace, at rates from about 1 to 35 requests per second):
| Metric | Improvement |
|---|---|
| Throughput vs standard SD | up to +14.76% (+8.32% average on the 7B/13B settings) |
| Latency vs standard SD | up to -20.18% |
| TTFT, with offloading (ablation) | -47.2% |
| vs DSD baseline (average) | +22.89% |
| vs BanditSpec (average) | +19.76% |
Tested on DeepSeek-R1-Distill-Qwen-7B (RTX 4090), Vicuna-13B (A100) and Qwen2.5-32B-Instruct (2x L20, tensor parallel).
The cost of switching speculation depth (and the KV cache layout that goes with it) ranges from 17.87 ms to over 102 ms depending on batch size and context length. The MAB planner factors this penalty into its arm selection, which prevents thrashing between speculation depths.
What if your draft model has a different vocabulary?
Every approach above requires the draft and target model to share a tokenizer. You can’t use a Llama-3 draft to accelerate a Gemma target because their vocabularies differ. Intel Labs and the Weizmann Institute addressed this with three algorithms published at ICML 2025 as an oral (arXiv:2502.05202).
The core idea behind SLEM (String-Level Exact Match): use plain text as a universal bridge between vocabularies.
- Draft model generates tokens in its own vocabulary
- Decode those tokens to a plain text string
- Re-tokenize that string into the target model’s vocabulary
- Target model verifies in a single parallel forward pass
- Accept the drafted text up to the first point where it disagrees with the target
This enables pairing drafts and targets that were previously incompatible. The paper’s benchmarks pair small drafters such as vicuna-68m and Qwen2.5-0.5B-Instruct with targets such as Mixtral-8x22B-Instruct and phi-4. SLEM achieves up to 2.8x speedup over autoregressive decoding on summarization, programming and long-context tasks.
Two additional algorithms cover different tradeoff points:
| Algorithm | How it works | Reported speedup | When to use |
|---|---|---|---|
| SLEM | Text-bridge between vocabularies | up to 2.8x | Maximum speed, any model pair |
| SLRS | String-level rejection sampling | Not benchmarked (theory only; cost grows quickly with token length) | Research interest |
| TLI | Draft samples only from the shared vocabulary subset | up to 1.7x | No text round trip; needs enough vocabulary overlap |
The authors merged SLEM and TLI into Hugging Face Transformers. For teams already using HF’s generate() pipeline, this is the lowest-friction path to cross-vocabulary speculative decoding.
Which approach fits your serving stack?
| Scenario | Best approach | Why |
|---|---|---|
| Single-user latency, dedicated GPU budget | Saguaro | 4.7x speedup at batch 1, needs separate draft GPU |
| Variable traffic, shared serving infra | Nightjar | Adapts to load, offloads draft when speculation hurts |
| Mismatched draft/target vocabularies | SLEM | Works across tokenizers |
| Production default, minimum integration effort | EAGLE-3 | Native in vLLM, SGLang, TensorRT-LLM |
| Maximum throughput on latest hardware | P-EAGLE | Up to 1.69x over EAGLE-3 on B200, vLLM v0.16.0+ |
Framework support at the time of writing (March 2026):
| Variant | vLLM | SGLang | TensorRT-LLM | HF Transformers |
|---|---|---|---|---|
| EAGLE-3 | Yes | Yes | Yes | Via speculators |
| P-EAGLE | Yes (v0.16+) | No | No | No |
| Saguaro | No (standalone repo) | No | No | No |
| Nightjar | No | No | No | No |
| SLEM/Universal | No | No | No | Yes (merged) |
| MTP (DeepSeek) | Yes | Yes | Yes | N/A |
| Medusa | Yes | No | Partial | Yes |
Saguaro and Nightjar are research implementations with standalone codebases (github.com/tanishqkumar/ssd for Saguaro). Check current framework release notes before assuming either is still standalone. For production, EAGLE-3 or P-EAGLE through vLLM is the pragmatic choice.
The practical composition: use EAGLE-3 as your production baseline, evaluate Saguaro if you have dedicated draft GPUs and need maximum single-user latency, and adopt SLEM when your best available draft model doesn’t share a tokenizer with your target. Nightjar’s load-adaptive idea is the one most likely to end up as a built-in serving feature rather than a system you deploy on its own.
For the serving infrastructure these techniques plug into, see LLM serving infrastructure.
Takeaways
- Speculative decoding is now production infrastructure, not a research trick. EAGLE-3 is supported in vLLM, SGLang and TensorRT-LLM.
- Saguaro (4.7x over AR at batch size 1) fills the draft model’s idle time during verification. The cost is a dedicated draft GPU, making it best for latency-critical deployments.
- Nightjar (up to +14.8% throughput and -20% latency versus fixed speculation) adapts speculation depth to server load. It addresses the real-world problem that fixed speculation hurts at high batch sizes.
- SLEM (up to 2.8x, ICML 2025) breaks the shared-vocabulary requirement by using text as the bridge between draft and target.
- Deloitte expects inference to be about two-thirds of AI compute in 2026. At low batch sizes, a 2x decode speedup can roughly halve the GPUs needed for latency-bound interactive traffic.
- For production today: start with EAGLE-3 in vLLM or SGLang. Evaluate Saguaro and SLEM as your requirements diverge from the default.