12 minute read

“In standard speculative decoding, the draft model sits idle while the target verifies. That idle time is the bottleneck Saguaro goes after.”

TL;DR

Three papers solve three different speculative decoding bottlenecks. Saguaro (ICLR 2026, Kumar, Dao and May) fills the draft model’s idle time during verification and reports 255.8 tok/s on Llama-3.1-70B at batch size 1, 4.7x over autoregressive decoding. Nightjar adapts speculation depth to server load, with up to 14.8% higher throughput and up to 20% lower latency than fixed-depth speculation. Intel and Weizmann’s SLEM algorithm lets a draft model accelerate a target with a different vocabulary, at up to 2.8x. Each targets a different deployment scenario, and understanding which one fits your serving stack is the point of this post. For the foundational inference techniques these build on, see 14 techniques for high-performance ML inference.

A chess board mid-game with several moves planned ahead shown as ghost pieces in transparent form at future positions

Why does speculative decoding matter more in 2026 than in 2024?

Because inference is now the dominant cost. Deloitte’s 2026 TMT Predictions expects inference to account for roughly two-thirds of all AI compute in 2026, up from about one-third in 2023, and the market for inference-optimized chips to exceed $50 billion in 2026.

Speculative decoding directly attacks this cost center. At low batch sizes, a 2x decode speedup lets the same GPUs serve about twice the latency-bound traffic. At batch size 1 (the latency-sensitive interactive case), standard speculative decoding already delivers 2-3x speedups. The three papers covered here push that envelope further: Saguaro to 4.7x, Nightjar to variable-load scenarios where fixed speculation hurts, and universal drafting to model pairs that were previously incompatible.

The production ecosystem has caught up too. EAGLE-3 (NeurIPS 2025) is supported in vLLM, SGLang and TensorRT-LLM. Amazon SageMaker AI picks EAGLE-2 or EAGLE-3 automatically based on model architecture. P-EAGLE, in vLLM since v0.16.0, adds parallel drafting, with vLLM reporting up to 1.69x over EAGLE-3 on B200. Speculative decoding moved from research novelty to production infrastructure between 2024 and 2026.

How does standard speculative decoding work?

A small draft model generates K candidate tokens. The large target model then verifies all K tokens in a single forward pass. Rejection sampling at each position ensures the output distribution is mathematically identical to sampling from the target alone. No quality loss.

Standard speculative decoding (one round):

Draft model (1B):   [t1] → [t2] → [t3] → [t4]     (fast, sequential)
                                                      |
Target model (70B):  ══════ verify all 4 in parallel ═══  (one forward pass)
                                                      |
Result:              [t1 ✓] [t2 ✓] [t3 ✗] → resample [t3*]
                     (if all 4 pass, the target adds a bonus token)

Why it works: at batch size 1 the target model’s forward pass costs nearly the same whether it processes 1 or K tokens, because it is memory-bandwidth bound. With K = 4 and a per-token acceptance rate between 0.6 and 0.8, the expected output is about 2.3 to 3.4 tokens per target call instead of 1.

Rough acceptance rates by task (illustrative; they vary widely by model pair, so measure on your own traffic):

Task type Acceptance rate (α)
Code, structured output 0.75-0.85
General chat, instructions 0.60-0.80
Creative writing, domain-specific 0.50-0.65

At low acceptance rates, the cost of drafting can outweigh the savings; where the break-even sits depends on how cheap the draft is relative to the target. For how large the draft should be, see SDSL’s scaling laws for speculative decoding. The three papers below each address a specific failure mode of the basic approach.

What does Saguaro change by speculating on speculations?

Saguaro (arXiv:2603.03251, ICLR 2026) is the paper’s optimized algorithm for “speculative speculative decoding” (SSD). It removes the sequential dependency between drafting and verification. In standard speculative decoding, the draft model sits idle during the entire verification phase. Saguaro fills that dead time.

sequenceDiagram
    participant D as Draft model (1B)
    participant T as Target model (70B)
    participant C as Speculation cache

    Note over D,T: Standard SD: draft idles during verify
    D->>T: Send draft tokens [t1..t4]
    Note over D: Idle waiting...
    T->>D: Verified: accept 3, bonus token t*

    Note over D,T: Saguaro: draft works during verify
    D->>T: Send draft tokens [t1..t4]
    D->>C: Predict likely outcomes (k=2,t*=A), (k=3,t*=B)
    D->>C: Generate continuations for each outcome
    T->>C: Verified: accept 3, bonus=B
    C->>T: Cache hit → instant next draft

The mechanics: while the target model verifies round T, the draft model predicts the most likely verification outcome as a (k, t) pair, where k is the number of accepted tokens and t is the bonus token. It generates continuations for several likely outcomes and stores them in a cache. When verification completes, a cache lookup returns the next draft with near-zero latency.

Three optimizations make this work:

  1. Geometric fan-out distributes the speculation budget across outcomes weighted by acceptance probability. The paper reports predicting the bonus token with up to 90% accuracy.
  2. Acceptance-cache tradeoff sampling downweights the draft’s top tokens, trading a little acceptance rate for a higher cache hit rate, with a tunable parameter.
  3. Adaptive fallback picks the backup speculator by batch size: a higher-quality fallback at small batches, a faster one above a critical batch size where overhead matters more.

Measured on Llama-3.1-70B Instruct (4x H100, tensor parallel) with Llama-3.2-1B as the draft on a separate H100, batch size 1, averaged over four datasets (paper appendix B.3):

Method Tokens/sec Speedup vs AR
Autoregressive 54.7 1.0x
Standard SD 161.8 3.0x
Saguaro 255.8 4.7x

The abstract’s headline is more conservative: about 30% faster on average than the strongest speculative decoding baselines, and up to 5x faster than autoregressive decoding in open-source engines.

The catch: Saguaro requires a dedicated GPU for the draft model, separate from the target model’s hardware. Standard SD often collocates draft and target on the same device. This is a deployment architecture change, not just a configuration flag.

How does Nightjar adapt to changing server load?

Nightjar (arXiv:2512.22420) solves a problem Saguaro ignores: what happens when your server load changes. Standard speculative decoding with a fixed speculation length helps at low request rates but hurts at high ones. The target model becomes compute-saturated at large batch sizes, and speculation adds overhead without benefit.

Nightjar uses a multi-armed bandit (MAB) planner that continuously adjusts the speculation depth γ ∈ {0, 1, …, Γmax} based on current batch size. Setting γ=0 disables speculation entirely. The planner uses batch size as its primary context signal and balances exploration and exploitation to find a good depth for each load level. The thresholds in the diagram below are illustrative; the planner learns them online.

graph TD
    A[Incoming requests] --> B{Current batch size?}
    B -->|Small| C[High γ<br/>Full speculation<br/>Max latency benefit]
    B -->|Medium| D[Medium γ<br/>Reduced speculation<br/>Balance throughput/latency]
    B -->|Large| E[γ = 0-1<br/>Minimal or no speculation<br/>Avoid overhead]
    E -->|Memory pressure| F[Offload draft model to CPU<br/>Reclaim GPU for KV cache]
    F -->|Load drops| D

When the server is heavily loaded and speculation is disabled (γ=0), Nightjar does something unusual: under GPU memory pressure it offloads the draft model to CPU and hands that memory to the KV cache, which allows larger batches. The paper reports that expanding the KV cache this way takes 143.9 ms and shrinking it back takes 11.9 ms with a Triton kernel.

Results under dynamic load (Poisson arrivals plus a segment of the Azure LLM inference trace, at rates from about 1 to 35 requests per second):

Metric Improvement
Throughput vs standard SD up to +14.76% (+8.32% average on the 7B/13B settings)
Latency vs standard SD up to -20.18%
TTFT, with offloading (ablation) -47.2%
vs DSD baseline (average) +22.89%
vs BanditSpec (average) +19.76%

Tested on DeepSeek-R1-Distill-Qwen-7B (RTX 4090), Vicuna-13B (A100) and Qwen2.5-32B-Instruct (2x L20, tensor parallel).

The cost of switching speculation depth (and the KV cache layout that goes with it) ranges from 17.87 ms to over 102 ms depending on batch size and context length. The MAB planner factors this penalty into its arm selection, which prevents thrashing between speculation depths.

What if your draft model has a different vocabulary?

Every approach above requires the draft and target model to share a tokenizer. You can’t use a Llama-3 draft to accelerate a Gemma target because their vocabularies differ. Intel Labs and the Weizmann Institute addressed this with three algorithms published at ICML 2025 as an oral (arXiv:2502.05202).

The core idea behind SLEM (String-Level Exact Match): use plain text as a universal bridge between vocabularies.

  1. Draft model generates tokens in its own vocabulary
  2. Decode those tokens to a plain text string
  3. Re-tokenize that string into the target model’s vocabulary
  4. Target model verifies in a single parallel forward pass
  5. Accept the drafted text up to the first point where it disagrees with the target

This enables pairing drafts and targets that were previously incompatible. The paper’s benchmarks pair small drafters such as vicuna-68m and Qwen2.5-0.5B-Instruct with targets such as Mixtral-8x22B-Instruct and phi-4. SLEM achieves up to 2.8x speedup over autoregressive decoding on summarization, programming and long-context tasks.

Two additional algorithms cover different tradeoff points:

Algorithm How it works Reported speedup When to use
SLEM Text-bridge between vocabularies up to 2.8x Maximum speed, any model pair
SLRS String-level rejection sampling Not benchmarked (theory only; cost grows quickly with token length) Research interest
TLI Draft samples only from the shared vocabulary subset up to 1.7x No text round trip; needs enough vocabulary overlap

The authors merged SLEM and TLI into Hugging Face Transformers. For teams already using HF’s generate() pipeline, this is the lowest-friction path to cross-vocabulary speculative decoding.

Which approach fits your serving stack?

Scenario Best approach Why
Single-user latency, dedicated GPU budget Saguaro 4.7x speedup at batch 1, needs separate draft GPU
Variable traffic, shared serving infra Nightjar Adapts to load, offloads draft when speculation hurts
Mismatched draft/target vocabularies SLEM Works across tokenizers
Production default, minimum integration effort EAGLE-3 Native in vLLM, SGLang, TensorRT-LLM
Maximum throughput on latest hardware P-EAGLE Up to 1.69x over EAGLE-3 on B200, vLLM v0.16.0+

Framework support at the time of writing (March 2026):

Variant vLLM SGLang TensorRT-LLM HF Transformers
EAGLE-3 Yes Yes Yes Via speculators
P-EAGLE Yes (v0.16+) No No No
Saguaro No (standalone repo) No No No
Nightjar No No No No
SLEM/Universal No No No Yes (merged)
MTP (DeepSeek) Yes Yes Yes N/A
Medusa Yes No Partial Yes

Saguaro and Nightjar are research implementations with standalone codebases (github.com/tanishqkumar/ssd for Saguaro). Check current framework release notes before assuming either is still standalone. For production, EAGLE-3 or P-EAGLE through vLLM is the pragmatic choice.

The practical composition: use EAGLE-3 as your production baseline, evaluate Saguaro if you have dedicated draft GPUs and need maximum single-user latency, and adopt SLEM when your best available draft model doesn’t share a tokenizer with your target. Nightjar’s load-adaptive idea is the one most likely to end up as a built-in serving feature rather than a system you deploy on its own.

For the serving infrastructure these techniques plug into, see LLM serving infrastructure.

Takeaways

  • Speculative decoding is now production infrastructure, not a research trick. EAGLE-3 is supported in vLLM, SGLang and TensorRT-LLM.
  • Saguaro (4.7x over AR at batch size 1) fills the draft model’s idle time during verification. The cost is a dedicated draft GPU, making it best for latency-critical deployments.
  • Nightjar (up to +14.8% throughput and -20% latency versus fixed speculation) adapts speculation depth to server load. It addresses the real-world problem that fixed speculation hurts at high batch sizes.
  • SLEM (up to 2.8x, ICML 2025) breaks the shared-vocabulary requirement by using text as the bridge between draft and target.
  • Deloitte expects inference to be about two-thirds of AI compute in 2026. At low batch sizes, a 2x decode speedup can roughly halve the GPUs needed for latency-bound interactive traffic.
  • For production today: start with EAGLE-3 in vLLM or SGLang. Evaluate Saguaro and SLEM as your requirements diverge from the default.