causal

LLMs Fail ARC Due to Novelty Not Parsing

LLMs fail ARC not because they cannot parse the small 2D grids (fine-tuning shows they encode solution programs fine for tasks they've seen), but because each new task requires synthesizing a new solution program on the fly, which is precisely what LLMs cannot do—they can only fetch memorized programs.

causalpending

Speaker

François Chollet

Evidence Quote

“You cannot memorize the solution programs in advance. You have to synthesize a new solution program on the fly for each new task. That's really what LLMs are struggling with.”

Source

Francois Chollet — Why the biggest AI models can't solve simple puzzles— Dwarkesh Patel Podcast
Created: 6/13/2026, 3:26:16 AM

Your notes

Loading notes...