factual

Benchmarks Follow Exponential Saturation Curve

Prior AGI-aspiring benchmarks like MMLU and MATH followed an almost exponential saturation curve—scoring 5%, 10%, 30%, 40% as compute increases across models, then shooting up to near-100%—and on HumanEval, upweighting cases where models occasionally get problems right (1/1000 to 1/100 to 1/10) leads to total saturation, raising the question of why ARC won't follow the same pattern.

factualpending

Speaker

Dwarkesh Patel

Evidence Quote

“Shouldn't we see the same pattern we saw across all these other benchmarks? You just eke out and then once you get the general idea, you just go all the way to a hundred?”

Source

Francois Chollet — Why the biggest AI models can't solve simple puzzles— Dwarkesh Patel Podcast
Created: 6/13/2026, 3:26:16 AM

Your notes

Loading notes...