causal
Benchmark Choice Determines What Scaling Optimizes
The way you measure performance is not a technical detail but the central issue: scaling laws relate compute to benchmark performance, and because nearly all LLM benchmarks are memorization-based (solvable by memorizing a finite set of reasoning patterns), scaling increases skill and usefulness but not intelligence—conflating skill with intelligence is the fundamental confusion.
causalpending
Speaker
François CholletEvidence Quote
“The way you measure performance is not a technical detail. It's not an afterthought because it's going to narrow down the set of questions that you're asking. Accordingly, it's going to narrow down the set of answers that you're looking for.”
Source
Francois Chollet — Why the biggest AI models can't solve simple puzzles— Dwarkesh Patel PodcastCreated: 6/13/2026, 3:26:16 AM
Your notes
Loading notes...