flash attention
An efficiency-improving algorithmic technique for computing attention in transformer models, cited as an example of software innovations that speed up effective compute.
As defined in Carl Shulman (Pt 1) — Intelligence explosion, primate evolution, robot doublings, & alignment
Where it comes up · 3
- AGI is still 30 years away — Ege Erdil & Tamay Besiroglu1 claim
- Carl Shulman (Pt 1) — Intelligence explosion, primate evolution, robot doublings, & alignment1 claim
- Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served1 claim
Most important claims · 3 of 3Search all claims for “flash attention” →
- Output (decode) being charged roughly 3-5x more than input (prefill) indicates that prefill is compute-limited while decode is memory-bandwidth-limited: processing many tokens in one pass amortizes the weight memory fetch across the pass length, so the per-token memory cost falls with longer passes, revealing the labs are heavily memory-bandwidth bottlenecked during decode.Dwarkesh Patel Podcast - Reiner Pope – The math behind how LLMs are trained and served
- Empirical estimates of the returns to research effort in software (across linear/SAT solvers, computer vision, RL, language modeling) are ambiguous about whether scaling cognitive effort produces accelerating growth or merely exponential growth; combined with the fact that software progress tracks hardware progress (~30%/year matching Moore's law, with the deep-learning acceleration coinciding with compute scaling), this argues against a hyperbolic faster-than-exponential software-only singularity.AGI is still 30 years away — Ege Erdil & Tamay Besiroglu
- The intelligence explosion feedback loop begins not at full human-level AI but when AI contributions to AI research become comparable to human contributions — boosting researcher productivity by 50-100% — because automating some tasks done in extreme profusion (like generating billions of programming problems or custom curricula) effectively doubles the research workforce even with the AI having weaknesses in other areas.Carl Shulman (Pt 1) — Intelligence explosion, primate evolution, robot doublings, & alignment