factual

ARC Benchmark Tests Adaptation to Novelty

ARC (Abstraction and Reasoning Corpus for AGI), released in 2019, is a benchmark of unique, original puzzles never seen before and partly kept private, designed so that memorization cannot help; humans solve them easily while deep learning and LLMs score very poorly—GPT-3 scored zero in the 2020 Kaggle competition—because each puzzle is genuinely novel and LLMs have essentially no ability to adapt to such novelty.

factualpending

Speaker

François Chollet

Evidence Quote

“deep learning methods and LLMs in particular have scored very poorly on ARC... people tried GPT-3, and it scored zero”

Source

280 | François Chollet on Deep Learning and the Meaning of Intelligence— Sean Carroll's Mindscape
Created: 6/13/2026, 12:07:39 AM

Your notes

Loading notes...