factual
ARC Benchmark Tests Adaptation to Novelty
ARC (Abstraction and Reasoning Corpus for AGI), released in 2019, is a benchmark of unique, original puzzles never seen before and partly kept private, designed so that memorization cannot help; humans solve them easily while deep learning and LLMs score very poorly—GPT-3 scored zero in the 2020 Kaggle competition—because each puzzle is genuinely novel and LLMs have essentially no ability to adapt to such novelty.
factualpending
Speaker
François CholletEvidence Quote
“deep learning methods and LLMs in particular have scored very poorly on ARC... people tried GPT-3, and it scored zero”
Source
280 | François Chollet on Deep Learning and the Meaning of Intelligence— Sean Carroll's MindscapeCreated: 6/13/2026, 12:07:39 AM
Your notes
Loading notes...