ARC benchmark
Abstraction and Reasoning Corpus: a set of IQ-test-like grid puzzles, each requiring inferring a transformation rule from a few input-output examples, designed to resist memorization and measure efficient skill acquisition.
As defined in Francois Chollet — Why the biggest AI models can't solve simple puzzles
Where it comes up · 2
- Francois Chollet — Why the biggest AI models can't solve simple puzzles10 claims
- Sarah Guo - The Power of Conviction - [Invest Like the Best, EP.383]1 claim
Most important claims · 10 of 11Search all claims for “ARC benchmark” →
- ARC functions as an IQ test for machine intelligence that, unlike most LLM benchmarks, is designed to resist memorization because each puzzle is novel and requires only 'core knowledge' (elementary physics, objectness, counting) that any young child possesses, rather than vast crammed knowledge.Francois Chollet — Why the biggest AI models can't solve simple puzzles
- Models are surpassing human benchmarks such as MMLU and legal or medical exams, so evaluation becomes difficult because human evaluation fails once models outperform experts, requiring new evaluation methods such as ARC or measuring human-plus-model performance.Sarah Guo - The Power of Conviction - [Invest Like the Best, EP.383]
- LLMs fail ARC not because they cannot parse the small 2D grids (fine-tuning shows they encode solution programs fine for tasks they've seen), but because each new task requires synthesizing a new solution program on the fly, which is precisely what LLMs cannot do—they can only fetch memorized programs.Francois Chollet — Why the biggest AI models can't solve simple puzzles
- In the original 2020 Kaggle competition for ARC (with a ~$20K prize), people tried GPT-3 on the public data shortly after its release and it scored zero, and the contest taught that there is no obvious shortcut to hack the benchmark.Francois Chollet — Why the biggest AI models can't solve simple puzzles
- The illegitimate way to win ARC is brute-forcing the space of possible tasks: generating hundreds of millions of programmatic ARC-task variations so that there is enough overlap with the test set to score highly via memorized fetching—this would reveal a benchmark flaw rather than genuine intelligence, whereas adding a code interpreter or fine-tuning on core knowledge is legitimate.Francois Chollet — Why the biggest AI models can't solve simple puzzles
- Average humans, tested via Amazon Mechanical Turk, scored about 85% on ARC, and any smart human can do 90-95% while even a five-year-old with little knowledge can do over 50%.Francois Chollet — Why the biggest AI models can't solve simple puzzles
- Dwarkesh's friends prompted Claude Opus to get 25% on the public ARC test by providing in-context examples that explain the reasoning of how one output was derived from another, expressing the JSON in a more tokenizer-amenable form, and using the code interpreter to write Python programs that produce the correct cell outputs.Francois Chollet — Why the biggest AI models can't solve simple puzzles
- A genuine ARC solution trained only on core knowledge, while not necessarily AGI itself, would be a huge milestone because it represents the ability to synthesize a generalizing problem-solving program from just two or three examples—an entirely new paradigm where you program by showing the computer the desired output rather than typing up the program yourself.Francois Chollet — Why the biggest AI models can't solve simple puzzles
- ARC gained little traction in the research community because benchmarks that gain traction are ones already fairly tractable—a group makes an initial breakthrough, attracting follow-up papers—whereas ARC is very hard for existing AI techniques and requires genuinely new ideas, so no initial breakthrough sparked the usual competitive dynamic.Francois Chollet — Why the biggest AI models can't solve simple puzzles
- The big labs tried their hand at ARC but, because they got bad results, did not publish anything—people only publish positive results—creating publication bias that obscures how poorly current techniques perform on the benchmark.Francois Chollet — Why the biggest AI models can't solve simple puzzles