normative
Brute-Forcing Task Space Defeats ARC's Purpose
The illegitimate way to win ARC is brute-forcing the space of possible tasks: generating hundreds of millions of programmatic ARC-task variations so that there is enough overlap with the test set to score highly via memorized fetching—this would reveal a benchmark flaw rather than genuine intelligence, whereas adding a code interpreter or fine-tuning on core knowledge is legitimate.
normativepending
Speaker
François CholletEvidence Quote
“That's defeating the purpose of the benchmark because then you can just solve it with that and you need to adapt just by fetching a memorized solution.”
Source
Francois Chollet — Why the biggest AI models can't solve simple puzzles— Dwarkesh Patel PodcastCreated: 6/13/2026, 3:26:16 AM
Your notes
Loading notes...