task distribution
The set of problems and task types used to evaluate AI systems, representing a sample of capabilities that may or may not be representative of all tasks humans perform
As defined in Understanding the Most Viral Chart in Artificial Intelligence | Odd Lots
Where it comes up · 1
Most important claims · 2 of 2Search all claims for “task distribution” →
- Joel suspects that METR's task distribution is increasingly becoming a narrower slice of all possible tasks and specifically overlaps more with the exact task distributions used by AI labs for training, meaning METR is measuring progress on tasks optimized for (rather than independent test of) lab capabilities.Understanding the Most Viral Chart in Artificial Intelligence | Odd Lots
- Joel acknowledges that METR's baseline methodology is imperfect and would ideally have 100x more resources (100 baselines per task, top-tier engineers, wider task distributions), but current limitations don't invalidate the core finding of exponential progress because doubling time is robust to baseline variability.Understanding the Most Viral Chart in Artificial Intelligence | Odd Lots