Speculative Decoding

Overview

Speculative decoding is an inference optimization technique that accelerates large language model generation by 2–3× without changing the model’s output quality. It works by using a small, fast “draft” model to propose multiple tokens ahead, then having the large “target” model verify all proposed tokens in a single forward pass. Accepted tokens are kept; rejected tokens are resampled from the target model.

The key insight is that LLM inference is fundamentally memory-bandwidth-bound, not compute-bound. Modern GPUs have massive parallel compute capacity that sits idle during standard autoregressive generation because each token depends on the previous one. Speculative decoding exploits this idle compute by giving the target model multiple tokens to verify simultaneously, rather than generating them one at a time.

For AI engineers, speculative decoding is one of the most impactful inference optimizations available today. Unlike quantization or pruning, it provides speedups with mathematically guaranteed identical outputs — the final text is exactly what the target model would have produced on its own. It’s now a production-standard technique supported natively by vLLM, TensorRT-LLM, and other serving frameworks.

How It Works

The Memory-Bandwidth Bottleneck

To understand why speculative decoding works, you need to understand the inference bottleneck. During autoregressive generation, each token requires loading the model’s entire weight matrix from GPU memory (HBM) into the compute cores. For a 70B-parameter model in FP16, that’s ~140GB of memory reads per token. Even on an H100 GPU with 3.35 TB/s memory bandwidth, this takes ~42ms per token — and the GPU’s 990 TFLOPS of compute capacity is barely utilized during this time.

Speculative decoding doesn’t reduce the memory reads — instead, it makes each read count for more by verifying multiple tokens per target-model forward pass.

The Algorithm Step by Step

Here’s how one speculative decoding iteration works:

1. Draft phase: The draft model autoregressively generates K tokens (typically K=3–8), producing both the tokens and their probability distributions q(x) at each position.
2. Verify phase: The target model processes the entire sequence (prompt + all K draft tokens) in a single forward pass, computing its own probability distributions p(x) at each position.
3. Accept/reject phase: For each draft token, from left to right, apply the rejection sampling criterion to decide whether to accept or reject it.
4. Resample if needed: At the first rejected position, sample a new token from an adjusted distribution derived from the target model. All tokens after the rejection point are discarded.
5. Guaranteed progress: Even if all K draft tokens are rejected, you still get at least one new token (the resampled one from the target model), so you never do worse than standard autoregressive decoding.

The Rejection Sampling Criterion

The acceptance logic is the mathematical core that guarantees output equivalence with the target model:

# For each draft token x_i at position i:
# q(x_i) = draft model's probability for token x_i
# p(x_i) = target model's probability for token x_i

# Accept with probability min(1, p(x_i) / q(x_i))
r = random.uniform(0, 1)
if r < min(1.0, p(x_i) / q(x_i)):
    accept(x_i)  # Token matches target model's distribution
else:
    # Reject and resample from adjusted distribution:
    # p_adjusted(x) = max(0, p(x) - q(x)) / sum(max(0, p(x') - q(x')))
    new_token = sample(adjusted_distribution)
    break  # Stop checking remaining draft tokens

This criterion has an elegant property: if the target model assigns equal or higher probability to a draft token than the draft model did, the token is always accepted. Tokens are only probabilistically rejected when the draft model was more confident than the target model, with the rejection probability proportional to the confidence gap.

Acceptance Rate: The Key Metric

The effectiveness of speculative decoding depends on the acceptance rate (α) — the fraction of draft tokens that the target model accepts. Higher acceptance rates mean more tokens are produced per verification step:

• α ≈ 0.8–0.9: Excellent. Most draft tokens are accepted, yielding near-optimal speedup. Typically achieved when the draft model is from the same family as the target (e.g., Llama 3 8B drafting for Llama 3 70B).
• α ≈ 0.5–0.7: Good. Meaningful speedups (1.5–2×), common with general-purpose draft models.
• α < 0.3: Poor. The draft model’s predictions diverge too much from the target, and most tokens are rejected. At this point, the overhead of running the draft model may negate the benefits.

The expected number of tokens generated per verification step is 1/(1-α) for geometric acceptance. With α=0.8, you get ~5 tokens per step on average.

Choosing a Draft Model

The draft model choice critically affects performance:

• Same family, smaller size: The best option when available. Llama 3 8B drafting for Llama 3 70B shares vocabulary, training data, and learned patterns, yielding high acceptance rates.
• Distilled models: A model specifically distilled from the target for the drafting task can achieve very high acceptance rates because it’s explicitly trained to match the target’s distribution.
• N-gram or retrieval-based drafters: For repetitive or template-heavy content, simple lookup-based drafters can be extremely fast with reasonable acceptance rates.
• Self-speculative decoding: Uses the target model’s own early layers or a subset of its attention heads as the drafter, eliminating the need for a separate model entirely.

Practical Applications

Production Deployment

Speculative decoding has matured from a research technique to a production standard:

• vLLM: Supports speculative decoding out-of-the-box with configurable draft models and speculation lengths. Enable it with --speculative-model and --num-speculative-tokens flags.
• TensorRT-LLM: NVIDIA’s inference engine includes optimized speculative decoding kernels that demonstrated 3.6× throughput improvements on H200 GPUs in benchmarks.
• Apple MLX: Includes speculative decoding for on-device inference on Apple Silicon, enabling faster local LLM generation on MacBooks and iPhones.
• Google DeepMind: Originally pioneered speculative decoding (called “speculative sampling”) and uses it extensively across their production serving infrastructure.

When Speculative Decoding Excels

• Latency-sensitive applications: Chatbots, autocomplete, and interactive coding assistants where time-to-first-token and tokens-per-second directly affect user experience.
• Long-form generation: Generating lengthy outputs (reports, articles, code) where the cumulative time savings of 2–3× speedup are substantial.
• Predictable content: Tasks with formulaic or structured outputs (JSON generation, template filling, code completion) where draft models achieve high acceptance rates.
• Single-request latency optimization: When you need to minimize latency for individual requests rather than maximize throughput across a batch of requests.

Limitations and Considerations

• Memory overhead: Running two models simultaneously requires additional GPU memory for the draft model’s weights and KV cache. For a 70B target + 7B draft, you need ~10% more memory.
• Batch size tradeoffs: At large batch sizes, the target model’s compute may already be well-utilized, reducing the benefit of speculative decoding. The technique is most beneficial for small batch sizes (1–8).
• Variable speedup: The actual speedup depends on the acceptance rate, which varies by prompt and task. Creative or reasoning-heavy tasks may see lower acceptance rates than factual or structured outputs.

Key Takeaways

• Speculative decoding uses a small draft model to propose tokens that a large target model verifies in parallel, achieving 2–3× speedup with mathematically identical outputs.
• The technique exploits the memory-bandwidth bottleneck in LLM inference — GPUs have idle compute during autoregressive generation that speculative decoding puts to use.
• Rejection sampling ensures the output distribution exactly matches the target model — this is a lossless optimization, not an approximation.
• The acceptance rate (α) is the critical metric: higher rates mean more tokens per verification step, with same-family draft models typically achieving α=0.8–0.9.
• Production frameworks (vLLM, TensorRT-LLM) include native support, making speculative decoding a straightforward deployment optimization.
• Best suited for latency-sensitive, small-batch serving scenarios; less beneficial at large batch sizes where compute is already well-utilized.