compute multiplier
An algorithmic improvement that effectively gets more performance per unit of compute, equivalent to having more compute.
As defined in Dwarkesh Patel Podcast - Eric Jang – Building AlphaGo from scratch
Other definitions
A technique or change that reduces the compute needed to reach a given level of model performance.
Dwarkesh Patel Podcast - The avalanche that hit math is coming for AI development
Where it comes up · 2
- Dwarkesh Patel Podcast - Eric Jang – Building AlphaGo from scratch3 claims
- Dwarkesh Patel Podcast - The avalanche that hit math is coming for AI development1 claim
Most important claims · 4 of 4Search all claims for “compute multiplier” →
- Individually good research ideas don't necessarily stack—training runs at AI labs have failed because of weird interactions between two seemingly good ideas pursued in parallel—which is why a single top-down vision of how the whole system should work matters, and why compute multipliers that have correlated benefits may not combine.Dwarkesh Patel Podcast - Eric Jang – Building AlphaGo from scratch
- The benefit of any given algorithmic compute-multiplier is likely transitory: many of KataGo's clever tricks (architecture choices, auxiliary supervision objectives, asynchronous distributed setups) matter much less on faster modern GPUs like Blackwell, and three years out they may stack even worse—so research taste is knowing how much the Bitter Lesson can buy at any given moment.Dwarkesh Patel Podcast - Eric Jang – Building AlphaGo from scratch
- In the fullness of time, compute is the single most important determinant of capability—scaling up energy, compute, and parameters inevitably yields intelligence and no algorithmic detail matters beyond that—but in the present day, finite compute forces researchers to use heuristics, which are likely redundant and transitory.Dwarkesh Patel Podcast - Eric Jang – Building AlphaGo from scratch
- Pretraining progress is extremely easy to verify because loss on a held-out test set is a direct measure, so GPT-7 could be RL-trained to minimize compute needed to reach a fixed loss, and swarms could search for the lowest perplexity.Dwarkesh Patel Podcast - The avalanche that hit math is coming for AI development