decode
To translate written words on a page into language; one component of reading often mistaken for the whole of reading ability.
As defined in Robert Pondiscio on How the Other Half Learns
Other definitions
The token-by-token generation phase of inference; in this framing it includes attention and feedforward network steps.
AI Market Jitters
Where it comes up · 5
- Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]4 claims
- Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served3 claims
- AI Market Jitters1 claim
- Robert Pondiscio on How the Other Half Learns1 claim
- Gavin Baker - Watts and Wafers - [Invest Like the Best, EP.473]1 claim
Most important claims · 10 of 10Search all claims for “decode” →
- Contrary to the misconception—driven partly by standardized tests—that reading is a decoding-plus-strategies skill that transfers to any text, reading comprehension depends overwhelmingly on shared background knowledge and vocabulary; the Recht and Leslie baseball study showed low-ability readers with high baseball knowledge outperformed high-ability readers lacking it, proving what you read about matters prodigiously.Robert Pondiscio on How the Other Half Learns
- Output (decode) being charged roughly 3-5x more than input (prefill) indicates that prefill is compute-limited while decode is memory-bandwidth-limited: processing many tokens in one pass amortizes the weight memory fetch across the pass length, so the per-token memory cost falls with longer passes, revealing the labs are heavily memory-bandwidth bottlenecked during decode.Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served
- RL training is less efficient per token than pre-training for two reasons: you may not train a backward pass on all rollouts (so FLOPs per token can be as low as 2 rather than the full forward+backward 6), and RL involves substantial decode which often runs at lower MFU than training, together giving roughly a 30% efficiency penalty.Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served
- Disaggregating prefill from decode opens new chip design trade-offs: prefill, which ingests the prompt and tracks state, is bound by memory capacity, while decode, which generates new tokens, is bound by memory bandwidth.Gavin Baker - Watts and Wafers - [Invest Like the Best, EP.473]
- Each pipeline hop from rack to rack adds latency (Pope estimates on the order of a few milliseconds, possibly an order of magnitude off) traversing GPU to network card to top-of-rack or data-center switch and back; because decode is sequential, these hop latencies stack across pipeline stages, so a handful of stages can add ~10ms per token, a meaningful fraction when a token takes ~20ms.Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served
- SRAM-based accelerators not constrained by HBM DRAM and built on older nodes can handle the feedforward network in disaggregated inference, which Gavin argues will improve AI ROI by splitting prefill, attention, and feedforward work across specialized chips.AI Market Jitters
- Lower decode latency shrinks wall clock time for long-horizon agent tasks, so a year-long compute build could take a month and a month-long build could take three days, accelerating scientific and technological progress.Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]
- Gavin's kernel work taught him that the math in decode is relatively easy and the real constraint is data movement across one chip or many, which motivated Etched's cluster-scale memory approach.Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]
- For decode, performance depends on memory bandwidth across the full scale-up cluster rather than on a single chip, because loading the model and KV cache faster allows more tokens per second per user, so the right question is cluster bandwidth rather than chip bandwidth.Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]
- Inference has two phases: prefill reads in large volumes of text to load the model's KV cache into the right state without predicting tokens, and decode uses that cache to generate output tokens, so Etched made separate technical bets for each phase.Etched - Building AI Hardware to Make Inference Faster and Cheaper - [Invest Like the Best, EP.480]