bitter lesson
The thesis that general methods leveraging more computation ultimately outperform approaches relying on human-engineered domain knowledge.
As defined in Dwarkesh Patel Podcast - Eric Jang – Building AlphaGo from scratch
Other definitions
Sutton's principle that general methods leveraging computation scale better than approaches relying on human-engineered knowledge; reframed here as finding techniques that most effectively and scalably use compute.
Some thoughts on the Sutton interviewIn machine learning, the principle that simpler, more general approaches that leverage raw compute and data scale outperform complex, engineered solutions incorporating domain knowledge
Your AI Stack Isn't Ready for Claude Mythos
Where it comes up · 7
- Don’t Believe AI Hype, This is Where it’s Actually Headed | Oxford’s Michael Wooldridge | AI History4 claims
- Richard Sutton – Father of RL thinks LLMs are a dead end2 claims
- Dwarkesh Patel Podcast - Eric Jang – Building AlphaGo from scratch2 claims
- Gavin Baker - Watts and Wafers - [Invest Like the Best, EP.473]2 claims
- Martin Casado - Entering Uncharted AI Territory - [Invest Like the Best, EP.381]2 claims
- Human vs. Machine Consciousness | Imperial’s Murray Shanahan1 claim
- Some thoughts on the Sutton interview1 claim
Most important claims · 10 of 14Search all claims for “bitter lesson” →
- A violation of Sutton's bitter lesson, in which human algorithmic ingenuity beats more compute, is the biggest risk to the AI trade, and near-ASI systems may make a temporary violation possible because they can make themselves more efficient.Gavin Baker - Watts and Wafers - [Invest Like the Best, EP.473]
- The benefit of any given algorithmic compute-multiplier is likely transitory: many of KataGo's clever tricks (architecture choices, auxiliary supervision objectives, asynchronous distributed setups) matter much less on faster modern GPUs like Blackwell, and three years out they may stack even worse—so research taste is knowing how much the Bitter Lesson can buy at any given moment.Dwarkesh Patel Podcast - Eric Jang – Building AlphaGo from scratch
- 'The Bitter Lesson' (article by Rich Sutton) argues that progress in AI has primarily come from willingness to throw more compute and data at problems rather than from clever algorithms or scientific insights; this is a sobering lesson suggesting that brute-force scaling, not theoretical breakthroughs, drives AI progress.Don’t Believe AI Hype, This is Where it’s Actually Headed | Oxford’s Michael Wooldridge | AI History
- Despite the bitter lesson of brute-force scaling, there is still 'magic' in AI—the fact that we can have conversations with systems like ChatGPT about virtually any topic, compared to five years ago when no such system existed, is astonishing and worth exploring deeply.Don’t Believe AI Hype, This is Where it’s Actually Headed | Oxford’s Michael Wooldridge | AI History
- Despite the 'bitter lesson,' there is still 'magic' in AI worth pursuing: understanding the principles governing how LLMs work, their capabilities and limitations, and the fundamental laws underlying these systems is now practical experimental science rather than pure philosophy—a watershed moment.Don’t Believe AI Hype, This is Where it’s Actually Headed | Oxford’s Michael Wooldridge | AI History
- Once the roughly 3,000 years of human-created data is exhausted, progress requires finding new ways to create data, and thousands of people writing code and annotating sentences will not match the scale of all code and text ever written.Martin Casado - Entering Uncharted AI Territory - [Invest Like the Best, EP.381]
- The bitter lesson is not about throwing away as much compute as possible, but about developing techniques that most effectively and scalably leverage compute.Some thoughts on the Sutton interview
- In the fullness of time, compute is the single most important determinant of capability—scaling up energy, compute, and parameters inevitably yields intelligence and no algorithmic detail matters beyond that—but in the present day, finite compute forces researchers to use heuristics, which are likely redundant and transitory.Dwarkesh Patel Podcast - Eric Jang – Building AlphaGo from scratch
- Rich Sutton's 'bitter lesson' shows that what works in AI is mindless scaling of computation, training data, and search rather than encoding human understanding into systems.Human vs. Machine Consciousness | Imperial’s Murray Shanahan
- In every historical case of the bitter lesson, you could start with human knowledge and then do scalable things, but in practice it has always turned out to be bad because people get locked into the human knowledge approach and are eventually superseded by methods that are truly scalable.Richard Sutton – Father of RL thinks LLMs are a dead end