Chinchilla
A DeepMind result showing optimal training requires scaling data (tokens) proportionally with model parameters, not just making models bigger.
As defined in Aravind Srinivas
Where it comes up · 2
- Aravind Srinivas1 claim
- Cursor Team1 claim
Most important claims · 2 of 2Search all claims for “Chinchilla” →
- OpenAI's GPT insight was that the transformer made language modeling roughly 100x more efficient at reaching the same performance than RNNs, so running the same compute yields a far better model; combined with scaling laws and the Chinchilla insight that you must scale both parameters and tokens (and token quality/dedup), this evolution — not attention alone — produced the breakthrough.Aravind Srinivas
- The original OpenAI scaling laws paper was slightly wrong due to issues with learning rate schedules, and Chinchilla later showed a more correct compute-optimal version, after which practitioners deviated again to optimize for inference budget rather than pure compute-optimality.Cursor Team