data distribution
The statistical spread of the data a model was trained on; a model performs well only on inputs within this distribution.
As defined in 280 | François Chollet on Deep Learning and the Meaning of Intelligence
Other definitions
The overall pattern and characteristics of data, including the frequency and relationships among different data points and features
Demis Hassabis — Scaling, superhuman AIs, AlphaZero atop LLMs, AlphaFold
Where it comes up · 4
- Dwarkesh Patel Podcast - The data black hole at the center of AI2 claims
- Demis Hassabis — Scaling, superhuman AIs, AlphaZero atop LLMs, AlphaFold1 claim
- 280 | François Chollet on Deep Learning and the Meaning of Intelligence1 claim
- Demis Hassabis — Scaling, superhuman AIs, AlphaZero atop LLMs, AlphaFold1 claim
Most important claims · 5 of 5Search all claims for “data distribution” →
- A critical research challenge is identifying and filling gaps in model training data distributions, including detecting bias, fairness issues, and underrepresented domains, then using synthetic generation to create data for those gaps.Demis Hassabis — Scaling, superhuman AIs, AlphaZero atop LLMs, AlphaFold
- Because LLMs are big differentiable parametric curves fit to a data distribution, you cannot input symbolic rules valid for whole categories of problems; you can only supply individual data points the model interpolates between, so fixing errors requires labor-intensive pointwise patches that address one query at a time rather than entire categories of issues.280 | François Chollet on Deep Learning and the Meaning of Intelligence
- It is not clear that AI labs have made much progress on training sample efficiency in recent years; instead, progress has come mainly from dramatically widening and improving the data distribution.Dwarkesh Patel Podcast - The data black hole at the center of AI
- The science of data curation and analysis—identifying gaps and holes in the data distribution and ensuring it represents the distribution to be learned—is still nascent; tricks like overweighting or replaying parts of data, or targeting synthetic generation at identified gaps, can address fairness, bias, and coverage.Demis Hassabis — Scaling, superhuman AIs, AlphaZero atop LLMs, AlphaFold
- How much out-of-distribution thinking white-collar jobs require depends on the nature of the job, since some roles like bank tellers and travel agents were automated long before modern AI, while others such as software engineering involve problems far from the training distribution.Dwarkesh Patel Podcast - The data black hole at the center of AI