speculative decoding
An inference technique where a small drafter model proposes multiple tokens and a big model verifies them at once, avoiding the single-token decode bottleneck.
As defined in Jeff Dean & Noam Shazeer — 25 years at Google: from PageRank to AGI
Other definitions
Inference optimization technique where multiple future tokens are predicted simultaneously or speculatively verified in parallel, reducing the number of sequential generation steps and speeding up text generation
The Engineering Unlocks Behind DeepSeek | YC Decoded
Where it comes up · 4
- The Engineering Unlocks Behind DeepSeek | YC Decoded2 claims
- This Tiny Engine Runs Impossibly Big AI Models Locally! (colibrì)1 claim
- Jeff Dean & Noam Shazeer — 25 years at Google: from PageRank to AGI1 claim
- Inference, Diffusion, World Models, and More | YC Paper Club1 claim
Most important claims · 5 of 5Search all claims for “speculative decoding” →
- An algorithmic improvement for inference efficiency is speculative decoding with drafter models: a small language model predicts several tokens (e.g., four) one at a time, then the big model checks all of them at once and advances past those it agrees with, turning a one-token-wide computation into a four-token-wide parallel computation and avoiding the single-token decode bottleneck, with the big model acting as a verifier.Jeff Dean & Noam Shazeer — 25 years at Google: from PageRank to AGI
- In speculative decoding, the key asymmetry that makes it work is that verification is easier than generation because the transformer architecture can compute probabilities for many tokens in parallel in one forward pass, while generation must be done autoregressive one token at a time.Inference, Diffusion, World Models, and More | YC Paper Club
- Colibri uses speculative decoding with a native multi-token prediction head that guesses several tokens ahead, enabling faster generation than one token per step.This Tiny Engine Runs Impossibly Big AI Models Locally! (colibrì)
- Multi-token prediction (MTP) modules can be repurposed for speculative decoding, reducing sequential processing steps and significantly speeding up generation during inference.The Engineering Unlocks Behind DeepSeek | YC Decoded
- Multi-token prediction modules in DeepSeek V3 can be repurposed for speculative decoding, reducing sequential processing steps and significantly speeding up generation during inferenceThe Engineering Unlocks Behind DeepSeek | YC Decoded