factual

Drafter models speed inference via verification

An algorithmic improvement for inference efficiency is speculative decoding with drafter models: a small language model predicts several tokens (e.g., four) one at a time, then the big model checks all of them at once and advances past those it agrees with, turning a one-token-wide computation into a four-token-wide parallel computation and avoiding the single-token decode bottleneck, with the big model acting as a verifier.

factualpending

Speaker

Jeff Dean

Evidence Quote

“you've basically been able to do a four-token width parallel computation instead of a one-token width computation in the big model.”

Source

Jeff Dean & Noam Shazeer — 25 years at Google: from PageRank to AGI— Dwarkesh Patel Podcast
Created: 6/13/2026, 3:30:41 AM

Your notes

Loading notes...