factual
Benchmarks Follow Exponential Saturation Curve
Prior AGI-aspiring benchmarks like MMLU and MATH followed an almost exponential saturation curve—scoring 5%, 10%, 30%, 40% as compute increases across models, then shooting up to near-100%—and on HumanEval, upweighting cases where models occasionally get problems right (1/1000 to 1/100 to 1/10) leads to total saturation, raising the question of why ARC won't follow the same pattern.
factualpending
Speaker
Dwarkesh PatelEvidence Quote
“Shouldn't we see the same pattern we saw across all these other benchmarks? You just eke out and then once you get the general idea, you just go all the way to a hundred?”
Source
Francois Chollet — Why the biggest AI models can't solve simple puzzles— Dwarkesh Patel PodcastCreated: 6/13/2026, 3:26:16 AM
Your notes
Loading notes...