Sheela
Sheela
Home
Add
Questions
Thinkers
Claims
People
Concepts
Jargon
Sources
Health
Predictions
Collections
Domain Landscape
Contribute
Review
Dedup
Topic requests
Settings
Help

← Jargon

Jargontechnology· 2 claims· 1 episode

task distribution

The set of problems and task types used to evaluate AI systems, representing a sample of capabilities that may or may not be representative of all tasks humans perform

As defined in Understanding the Most Viral Chart in Artificial Intelligence | Odd Lots


Where it comes up · 1
  • Understanding the Most Viral Chart in Artificial Intelligence | Odd Lots2 claims

Most important claims · 2 of 2Search all claims for “task distribution” →
  • Joel suspects that METR's task distribution is increasingly becoming a narrower slice of all possible tasks and specifically overlaps more with the exact task distributions used by AI labs for training, meaning METR is measuring progress on tasks optimized for (rather than independent test of) lab capabilities.Understanding the Most Viral Chart in Artificial Intelligence | Odd Lots
  • Joel acknowledges that METR's baseline methodology is imperfect and would ideally have 100x more resources (100 baselines per task, top-tier engineers, wider task distributions), but current limitations don't invalidate the core finding of exponential progress because doubling time is robust to baseline variability.Understanding the Most Viral Chart in Artificial Intelligence | Odd Lots