scaling laws
Empirical relationships showing AI model performance improves predictably with more data, parameters, and compute.
As defined in lex_demis-hassabis-2
Other definitions
Empirical observations that larger language models (more parameters, more training data, more compute) consistently achieve better performance across tasks, often following predictable power-law relationships
How ChatGPT changed everything: Thoughts from the frontline of the AI/ML revolutionEmpirically observed relationships showing how AI model performance improves predictably as more compute, data, and parameters are added; foundational to the hypothesis that continued scaling will lead to AGI.
Week 1 of the Trump administration, AGI timelines, and DeepSeek
Where it comes up · 61
- Anthropic CEO Dario Amodei Talks Scaling Laws, AI Arms Races, and Radical Abundance9 claims
- Dario Amodei (Anthropic CEO) — The hidden pattern behind every AI breakthrough5 claims
- Gavin Baker - AI, Semiconductors, and the Robotic Frontier - [Invest Like the Best, EP.385]5 claims
- Capital, Compute & AI Scaling4 claims
- Dan Sundheim - The Art of Public and Private Market Investing - [Invest Like the Best, EP.460]4 claims
- Sam Altman - How to Make an Abundant Future - [Invest Like the Best, EP.484]4 claims
- AI Trends 2025: Mary Meeker Deck Deep Dive Part 13 claims
- Carl Shulman (Pt 1) — Intelligence explosion, primate evolution, robot doublings, & alignment3 claims
and 53 more
Most important claims · 10 of 117Search all claims for “scaling laws” →
- Gwern foresaw scaling not via a eureka moment but by repeatedly noticing, over years, that dataset sizes, model sizes, and GPU counts kept growing while neural networks kept generalizing to ever-broader use cases, gradually shifting his prior away from the view that algorithms and deep insight are paramount toward the view that intelligence is mostly compute applied to data and parameters.Gwern — Anonymous writer who predicted AI trajectory on $12K/year salary
- Scaling laws in neural networks are empirically demonstrated as deep and fundamental truths, yet lack complete theoretical explanation—neural networks designed in the 1940s still show consistent capability improvements with increasing compute without hitting a performance wall.OpenAI's Greg Brockman: Why Human Attention Is the New Bottleneck
- Scaling matters profoundly for AI capabilities; there are predictable scaling laws showing smooth exponential relationships between compute/data and capability across many different axes of variation, representing a deep scientific discovery about the nature of intelligence.Sam Altman and Greg Brockman on AI and the Future (Full Audio)
- Automated researchers could develop a much better science of learning from small experiments, extending scaling laws and discovering phenomena like muP, which predicts optimal hyperparameters for larger runs from models 100x smaller, and GPT-4's loss was predicted from models trained with 10,000x less compute.Dwarkesh Patel Podcast - The avalanche that hit math is coming for AI development
- Under the Chinchilla scaling constants, where parameter and data loss terms add independently, even infinitely many parameters would reduce the data needed for a given loss by only about a factor of 10, and the result would not change with different constants, so scaling cannot close a gap of thousands to millions of times.Dwarkesh Patel Podcast - The data black hole at the center of AI
- Eroom's Law (Moore's Law spelled backward) describes the consistent decrease since the 1950s in the number of new medicines invented per billion dollars invested in biopharma, persisting across many technological transitions—an inverse of compute's Moore's Law trajectory.Evolution designed us to die fast; we can change that — Jacob Kimmel
- Scaling laws show that when you combine data, compute, and model size as ingredients in proportion to each other—similar to a chemical reaction—you get intelligence as an output, and this has been validated repeatedly with models like GPT-2 starting in 2019.The AI Tsunami is Here & Society Isn't Ready | Dario Amodei x Nikhil Kamath | People by WTF
- Because model quality was improving exponentially and cost could keep falling, demand for AI at sufficiently high capability and low price was basically uncapped, so no matter how efficient algorithms became, the company would always want more compute to turn electricity into useful intelligence.Sam Altman - How to Make an Abundant Future - [Invest Like the Best, EP.484]
- The reporting that AI progress had stalled was really that models were not improving relative to expectations or the amount of compute applied, not that models were not getting better, so labs will keep trying to find the unlock.Capital, Compute & AI Scaling
- People writing about AI around 2020 missed scaling because they hadn't appreciated prior scaling results (e.g. AlphaZero discovered partly via Bayesian hyperparameter optimization that revealed tree search could be reduced, and Baidu's 2017 scaling-laws paper that showed scaling laws persist), and because they shared the error of believing algorithms matter more than compute.Gwern — Anonymous writer who predicted AI trajectory on $12K/year salary