causal
LLMs Fail ARC Due to Novelty Not Parsing
LLMs fail ARC not because they cannot parse the small 2D grids (fine-tuning shows they encode solution programs fine for tasks they've seen), but because each new task requires synthesizing a new solution program on the fly, which is precisely what LLMs cannot do—they can only fetch memorized programs.
causalpending
Speaker
François CholletEvidence Quote
“You cannot memorize the solution programs in advance. You have to synthesize a new solution program on the fly for each new task. That's really what LLMs are struggling with.”
Source
Francois Chollet — Why the biggest AI models can't solve simple puzzles— Dwarkesh Patel PodcastCreated: 6/13/2026, 3:26:16 AM
Your notes
Loading notes...