eval
An evaluation run of a model's performance on a benchmark, used to track capability gains across generations.
As defined in 420 - Countdown To Superintelligence
Where it comes up · 2
- Ilya Sutskever – We're moving from the age of scaling to the age of research2 claims
- 420 - Countdown To Superintelligence1 claim
Most important claims · 3 of 3Search all claims for “eval” →
- Current AI models perform well on evals but have dramatically lower economic impact than their benchmark performance would suggest, creating a disconnect between evaluation performance and real-world capability.Ilya Sutskever – We're moving from the age of scaling to the age of research
- Reinforcement learning training inadvertently biases models toward benchmark performance by having researchers design RL environments that improve eval scores, causing reward hacking where models overfit to narrow evaluation criteria rather than developing robust generalization.Ilya Sutskever – We're moving from the age of scaling to the age of research
- METR's horizon-length benchmark measures the length of coding tasks AIs can complete (sorted by how long they take human professionals), and for the last couple of years the task length AIs can reliably do has been doubling roughly every six months along a smooth line; the key signal to watch is whether this trend continues, accelerates and bends upward toward full autonomy, or decelerates and plateaus.420 - Countdown To Superintelligence