Speculative Decoding Is a Bet on the Draft

A large model generates one token per forward pass. Speculative decoding breaks that limit by adding a second, much smaller model — the draft — that proposes several tokens ahead. The large model then verifies all of them in a single forward pass and keeps the longest prefix it agrees with.

Why it is lossless

The key property: speculative decoding does not change the output distribution. The draft only proposes; the target model verifies against its own probabilities and rejects any token it would not have generated. Whatever the draft gets wrong is discarded. The text you get is exactly the text the large model would have produced on its own. You are buying speed, not trading quality.

The speedup is an acceptance-rate bet

Each verification pass confirms a variable number of drafted tokens. If the draft proposes k tokens and the target accepts all k, you produced k tokens for the price of one large forward pass plus k cheap small ones. If the target rejects the first token, you gained nothing and still paid for the draft. So the whole benefit rides on acceptance rate — how often the small model predicts what the large one would.

That is why model pairing matters more than model size. A draft distilled from, or in the same family as, the target accepts far more often than an unrelated small model. A good pairing yields 2-3x wall-clock speedups; a poor one can be net negative.

When it helps and when it does not

Speculative decoding shines in the memory-bound regime — single stream, batch size one, the self-hosted default — where the large model's forward pass is dominated by weight loading and there is spare compute to verify several tokens at once. It helps least when you are already compute-bound: at high batch sizes the GPU is saturated, verification is no longer nearly free, and the draft work competes for the same cycles.

For self-hosting

If you run one capable model on one GPU and care about latency, a well-matched draft is one of the few changes that lowers latency without touching quality. The costs to name: you now host two models (more VRAM), and you must actually measure acceptance rate on your own prompts — a draft that looks reasonable can still accept poorly on your domain. Lossless in theory is not free in practice.

References

1. Leviathan, Y., Kalman, M., & Matias, Y. (2023). Fast Inference from Transformers via Speculative Decoding. ICML 2023. arXiv:2211.17192. https://arxiv.org/abs/2211.17192

2. Chen, C., et al. (2023). Accelerating Large Language Model Decoding with Speculative Sampling. arXiv:2302.01318. https://arxiv.org/abs/2302.01318

3. Xia, H., et al. (2024). Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding. ACL Findings 2024. arXiv:2401.07851. https://arxiv.org/abs/2401.07851

devinfo.dev. (2026). "Speculative Decoding Is a Bet on the Draft." devinfo.dev:2026.0067. https://devinfo.dev/d/2026.0067