causal

Benchmark Choice Determines What Scaling Optimizes

The way you measure performance is not a technical detail but the central issue: scaling laws relate compute to benchmark performance, and because nearly all LLM benchmarks are memorization-based (solvable by memorizing a finite set of reasoning patterns), scaling increases skill and usefulness but not intelligence—conflating skill with intelligence is the fundamental confusion.

causalpending

Speaker

François Chollet

Evidence Quote

“The way you measure performance is not a technical detail. It's not an afterthought because it's going to narrow down the set of questions that you're asking. Accordingly, it's going to narrow down the set of answers that you're looking for.”

Source

Francois Chollet — Why the biggest AI models can't solve simple puzzles— Dwarkesh Patel Podcast
Created: 6/13/2026, 3:26:16 AM

Your notes

Loading notes...