parameter count
The total number of learnable weights and biases in a neural network; a measure of model size and capacity, often expressed in millions (M), billions (B), or trillions (T) of parameters.
As defined in Breakthrough potential of AI | Sam Altman | MIT 2023
Where it comes up · 3
- Breakthrough potential of AI | Sam Altman | MIT 20233 claims
- The AI Reset Is Here: Search, Jobs, and Everything Else w/ Anish Acharya, Dave Blundin, Salim Ismail1 claim
- Jeff Dean (Google): Exciting Trends in Machine Learning1 claim
Most important claims · 5 of 5Search all claims for “parameter count” →
- There is a meaningful distinction between having scaling continue at large parameter counts (moving from 1 trillion to 10 trillion to 10 quadrillion parameters) versus achieving surprising emergent capabilities where the algorithm does things that weren't anticipated purely through scale.Breakthrough potential of AI | Sam Altman | MIT 2023
- Large language model parameter counts have grown exponentially: GPT-2 had 1.5 billion parameters (2019), T5 had 11 billion, GPT-3 had significantly more, and Gemini (co-led by the speaker) continues this progression with multiple sizes including Ultra, Pro, and Nano variants optimized for different deployment scenarios.Jeff Dean (Google): Exciting Trends in Machine Learning
- Parameter count is not the right metric to focus on for advancing AI capability; the gigahertz race in chips during the 1990s-2000s is an instructive analogy—manufacturers pointed to large numbers (gigahertz) that became irrelevant to user experience, so AI should focus on capability and safety rather than parameter scaling.Breakthrough potential of AI | Sam Altman | MIT 2023
- Parameter count is becoming a less useful metric for comparing model capability as reasoning models and training techniques improve, making it harder for researchers to distinguish between raw model intelligence and reasoning capability improvements.The AI Reset Is Here: Search, Jobs, and Everything Else w/ Anish Acharya, Dave Blundin, Salim Ismail
- The questioner built neural networks from sub-1-million parameter models, watched them scale to billions (GPT-2 ~1.5B), then to 175 billion (GPT-3), and observed mind-blowing differences in capability at each order of magnitude increase.Breakthrough potential of AI | Sam Altman | MIT 2023