Francois Chollet — Why the biggest AI models can't solve simple puzzles
What this covers
François Chollet sits with Dwarkesh Patel to argue that large language models possess skill without genuine intelligence, and that scaling alone cannot produce artificial general intelligence. The core distinction turns on Skill vs Intelligence: LLMs excel through memorization and interpolation within their training distribution, but lack the adaptive capacity to synthesize novel solutions for unseen problems. Chollet designed the ARC benchmark specifically to expose this gap. The conversation moves from defining what intelligence actually is—the ability to acquire new capabilities efficiently and handle novelty on the fly—through concrete demonstrations of where LLMs fail, to a proposal for the path forward: merging System 1 / System 2 cognition, combining deep learning's efficiency with discrete program synthesis.
The range spans technical architecture to human development. Chollet examines why test time fine-tuning masks LLM incapacity (performance collapses from 35% to 1–2% without it), why a transformer trained on the Bay Area cannot generalize to London driving conditions, and why LLMs solve Caesar ciphers only for memorized shift values but fail on arbitrary ones. He contrasts the data-efficiency of program synthesis against the compute-efficiency of deep learning, explaining the complementary strengths that motivate a hybrid approach. The discussion also touches how much intelligence is genetic versus learned, when core knowledge crystallizes in human development, and whether software engineering will shrink or grow in a world of genuine AI adaptation. An alternative hypothesis—that intelligence reduces to hierarchical memory—receives respectful attention before Chollet clarifies why mere association falls short when faced with truly novel tasks.
Chollet argues that LLMs achieve skill through memorization and interpolation but lack genuine intelligence—the ability to synthesize novel programs on the fly to adapt to unseen problems—and that scaling alone will not produce AGI; ARC is designed to expose this gap and the path forward is merging deep learning with discrete program synthesis.
- ARC is engineered to resist memorization, so each puzzle requires on-the-fly adaptation rather than fetching a stored solution program
- Skill is not intelligence: scaling a memorization-based system increases usefulness across tasks but not generality
- The way forward is a hybrid System 1 (deep learning intuition) + System 2 (discrete program search) architecture
This asset isn't compiled yet
You're seeing its claims, ranked. Compile it to build the argument threads, weight them, and check each claim against your library — the full view.
General intelligence is not task-specific skill scaled up to many skills, because the space of possible skills is infinite; rather, generality is the ability to approach any arbitrary problem and quickly master it using very little data, which fundamentally requires the capacity to adapt and learn on the fly.
“general intelligence is not task-specific skill scaled up to many skills, because there is an infinite space of possible skills.”
Genuine intelligence is the ability to adapt to novelty on the fly and acquire new skills efficiently, which differs fundamentally from memorization (reciting what one has seen) because the world constantly changes and the space of possible tasks is infinite, so no system can be pre-trained on everything it will encounter at test time.
“The big difference is that you can never pre-train on everything that you might see at test time because the world changes all the time.”
Humans developed general intelligence rather than hardcoded behavioral programs because the environment is not static and predictable; if the world were a fixed distribution, evolution would have written a perfect hardcoded connectome into our genes (as it did for many insects), but instead we are born with little knowledge and the ability to learn and adapt efficiently.
“If our environment were static and predictable enough, what would have happened is that evolution would have found the perfect behavioral program: a hard-coded, static behavioral program.”
All human thought, even seemingly pure reasoning like proving a mathematical theorem, is a mixture of interpolated memorization-based Type 1 thinking and Type 2 reasoning; humans are never at a pure end of the pattern-matching-versus-reasoning spectrum but always combine discrete search with intuition guided by previously seen proof shapes.
“All of our thoughts, everything we do, is a mixture of interpolated memorization-based thinking, Type 1 thinking, and Type 2 thinking.”
Grokking—the transition from a memorized, overfit input-output mapping to a generalizing solution—is an instance of the minimum description length principle: the program that generalizes best is the shortest, achieved by progressively pruning and compressing the mapping, so it is memorization plus regularization rather than a separate mechanism.
“It's basically an instance of the minimum description length principle. Given a problem, you can just memorize a pointwise input-to-output mapping, which is completely overfit.”
The way you measure performance is not a technical detail but the central issue: scaling laws relate compute to benchmark performance, and because nearly all LLM benchmarks are memorization-based (solvable by memorizing a finite set of reasoning patterns), scaling increases skill and usefulness but not intelligence—conflating skill with intelligence is the fundamental confusion.
“If you look at the benchmarks we are using for LLMs, they are all memorization-based benchmarks.”
ARC functions as an IQ test for machine intelligence that, unlike most LLM benchmarks, is designed to resist memorization because each puzzle is novel and requires only 'core knowledge' (elementary physics, objectness, counting) that any young child possesses, rather than vast crammed knowledge.
“ARC is intended as a kind of IQ test for machine intelligence. What makes it different from most LLM benchmarks out there is that it’s designed to be resistant to memorization.”
Jack Cole's state-of-the-art LLM approach to ARC (35% with a 240M-parameter model) works only because of test-time fine-tuning—on-the-fly fine-tuning a version of the LLM for each test problem—which adds the active inference LLMs normally lack; without it, performance collapses to a negligible 1-2%.
“What Jack Cole is actually doing is that for every test problem, it’s on-the-fly fine-tuning a version of the LLM for that task. That's really what's unlocking performance.”
Deep learning (gradient descent) is compute-efficient but data-inefficient and limited to generalizing within its data distribution, while discrete program search (combinatorial search) is extremely data-efficient—learning a generalizable program from one or two examples—but compute-inefficient due to combinatorial explosion; every limitation of one is a strength of the other, so the path forward is to merge them.
“Gradient descent is very compute efficient... but it is very data inefficient... Meanwhile, if you look at discrete program search... This process is extremely data efficient... The big limitation is that it's extremely compute inefficient because you're running into combinatorial explosion”
Deep learning is intrinsically limited to local generalization because the substrate of the model is a big parametric curve; to achieve broad or extreme generalization you must move to a different model type—discrete program search/program synthesis—where the model is a discrete graph of operators learned via combinatorial search.
“It's intrinsically limited because the substrate of your model is a big parametric curve. All you can do with this is local generalization.”
An alternative hypothesis (attributed to Trenton Bricken) holds that intelligence is just hierarchically associated memory: a figure like Sherlock Holmes is sample-efficient at a crime scene because he has learned higher-level associations, so program synthesis in the brain reduces to querying the right circuit through connected synapses—making intelligence memory in a fundamental sense.
“Former guest Trenton Bricken advanced an alternative hypothesis that intelligence is just hierarchically associated memory.”
Differences in intelligence between humans are mostly genetic in nature, which means no amount of training data can turn an unintelligent person into Einstein; this implies that for AI, a better architecture or algorithm is required, not merely more training data.
“There is extensive evidence that differences in intelligence are mostly genetic in nature. That means that if you take someone who is not very intelligent, there is no amount of training data you can expose that person to that would make them become Einstein.”
LLMs are big interpolative memories—parametric curves fitted to a data distribution—that maintain a bank of memorized solution programs; when given a puzzle they fetch and apply the appropriate stored program, which looks like reasoning but is program fetching, not on-the-fly program synthesis.
“LLMs are very good at memorizing small static programs. They've got this sort of bank of solution programs. When you give them a new puzzle, they can just fetch the appropriate program and apply it. It looks like reasoning but it's not really doing any sort of on-the-fly program synthesis. All it's doing is program fetching.”
Reasoning has two definitions: (1) identifying a memorized program template, plugging in new values, and running it—which LLMs do—and (2) when no program exists in memory, synthesizing a new program on the fly from bits and pieces of existing programs, which is dramatically harder and what true reasoning requires.
“When you're faced with a puzzle and you don't already have a program in memory to solve it, it’s the ability to synthesize on the fly a new program based on bits and pieces of existing programs that you have... That's actually dramatically harder than just fetching the right memorized program and reapplying it.”
With memorization you can automate almost anything—potentially generating trillions of dollars of economic value and replacing many jobs—as long as the task is a static distribution that can be precisely described in advance; but automation is not the same as intelligence, and intelligence is only required the moment you must deal with change, novelty, and uncertainty.
“with memorization you can automate almost anything as long as it's a static distribution, as long as you don't have to deal with change.”
Memory and intelligence are separate components: scaling up the interpolative memory is the easy part (just a big curve needing more data), while the hard, non-obvious part is the architecture of intelligence (System 2); we currently have the memory but not the intelligence, though an intelligence not hooked up to extensive memory would also be of limited use.
“Memory and intelligence are separate components. We have the memory. We don't have the intelligence yet.”
LLMs fail ARC not because they cannot parse the small 2D grids (fine-tuning shows they encode solution programs fine for tasks they've seen), but because each new task requires synthesizing a new solution program on the fly, which is precisely what LLMs cannot do—they can only fetch memorized programs.
“The reason LLMs don't do well on ARC is really just the unfamiliarity aspect. Each new task is different from every other task.”
Core knowledge is mostly acquired through experience very early—within the first 3-4 years of life rather than in school—such that by age four a person has all the core knowledge they will need as an adult; humans are not blank slates and are born with a small amount of innate world knowledge, but most core knowledge is learned.
“It's actually acquired very early in the first 3-4 years of your life. By age four, you have all the core knowledge you're going to need as an adult.”
In five years there will be more software engineers than there are today, not fewer, because software development inherently involves dealing with novelty all the time, and if LLMs could truly generalize to develop software for problems they've never seen, there would be no software engineers already.
“In five years, there will be more software engineers than there are today, not fewer.”
In the original 2020 Kaggle competition for ARC (with a ~$20K prize), people tried GPT-3 on the public data shortly after its release and it scored zero, and the contest taught that there is no obvious shortcut to hack the benchmark.
“This was right around the same time as GPT-3 was released. People of course tried GPT-3 on the public data. It scored zero.”
LLMs can solve a Caesar cipher for common shift values like three or five (which appear frequently in internet examples) but fail for an arbitrary value like nine, proving they have memorized specific cases rather than the generalized algorithm—if they could synthesize the solver on the fly, the shift value would not matter since it doesn't increase problem complexity.
“they can do it for a transposition length of like three or five, because those are very common numbers in examples provided on the internet. If you try to do it with an arbitrary number like nine, it's going to fail.”
OpenAI set back progress toward AGI by roughly 5-10 years for two reasons: it triggered the complete closing-down of frontier research publishing, and it sparked a burst of LLM hype that sucked the oxygen out of the room so that nearly everyone now does LLMs—which Chollet views as an off-ramp on the path to AGI—diverting resources from more diverse exploratory directions.
“OpenAI basically set back progress towards AGI by quite a few years, probably like 5-10 years.”
Bigger models are more sample-efficient because they have more reusable building blocks to lean on when picking up new patterns; however, this efficiency only holds to the extent that new patterns match what the model already learned, and it fails entirely on genuinely novel inputs outside the steady distribution, like ARC puzzles.
“Because they have more reusable building blocks that they can lean on to pick up new patterns in their training data.”
Self-driving cars, which use the same transformer architecture as LLMs, cannot generalize—a system trained in the Bay Area cannot be dropped into New York or London (where people drive on the left), and cannot even generalize to a new city without environment-specific training—demonstrating these models require training on each specific environment rather than adapting to novelty.
“You take a self-driving car operating in the Bay Area. Do you think you could just drop it in New York City or drop it in London, where people drive on the left? No, it's going to fail.”
The way forward is a hybrid system whose outer structure is discrete program search (System 2) but which fixes combinatorial explosion using deep learning (System 1) as an intuition engine—using deep learning models to provide the shape of solutions, suggest the most likely next steps in program space, and give feedback on whether to backtrack—producing an on-the-fly synthesis engine that assembles a generalizable model from very little data.
“The outer structure is going to be a discrete program search system. You're going to fix the fundamental limitation of discrete program search, which is combinatorial explosion, with deep learning. You're going to leverage deep learning to guide and to provide intuition in program space, to guide the program search.”
Two ARC approaches succeed at opposite ends of one spectrum: the LLM approach (Jack Cole) uses an enormous DSL of millions of vector programs mined by pre-training with very shallow recombination, while pure discrete program search uses a tiny DSL of 100-200 primitives with very deep, sophisticated search; the best value for compute, and the most progress toward near-term AGI, lies in merging them—building a richer learned bank of primitives plus some depth of search.
“You have an extraordinarily large DSL and the fine-tuning is very shallow recombination of these primitives. Discrete program search is very deep recombination with a very small set of primitive programs. The LLM approach is the same but on the complete opposite end of that spectrum.”
Architectural improvements are in extremely low-hanging-fruit territory, as evidenced by Gemini 1.5 Flash performing as well as GPT-4 did at its release a year earlier but at 57 times cheaper output, indicating part of the scaling story is rapid efficiency gains from architecture rather than raw size alone.
“A model like Gemini 1.5 Flash performs as well as GPT-4 did when GPT-4 was released a year ago, but is 57 times cheaper on output.”
The illegitimate way to win ARC is brute-forcing the space of possible tasks: generating hundreds of millions of programmatic ARC-task variations so that there is enough overlap with the test set to score highly via memorized fetching—this would reveal a benchmark flaw rather than genuine intelligence, whereas adding a code interpreter or fine-tuning on core knowledge is legitimate.
“The part that would be cheating is trying to anticipate what might be in the test, like brute force the space of possible tasks and train a memorization system on that.”
Prior AGI-aspiring benchmarks like MMLU and MATH followed an almost exponential saturation curve—scoring 5%, 10%, 30%, 40% as compute increases across models, then shooting up to near-100%—and on HumanEval, upweighting cases where models occasionally get problems right (1/1000 to 1/100 to 1/10) leads to total saturation, raising the question of why ARC won't follow the same pattern.
“you see this almost exponential curve. It gets 5%, 10%, 30%, 40% as you increase the compute across models, and then it just shoots up.”
Average humans, tested via Amazon Mechanical Turk, scored about 85% on ARC, and any smart human can do 90-95% while even a five-year-old with little knowledge can do over 50%.
“Not really, we actually tried with average humans. They scored about 85.”
Gemini 1.5's ability to learn a low-resource language (under 200 speakers) in-context from a grammar book and dictionary is not on-the-fly synthesis but the model mining a required template from its vast training data and reusing it, since LLMs have a very poor ability to synthesize or adapt new program templates on the fly.
“I would assume that it has simply mined from its extremely extensive, unimaginably vast training data. It has mined the required template and then it's just reusing it.”
Around 2015-2016 there were roughly a thousand times fewer people doing AI, yet the rate of progress was higher because people explored more directions and the field felt open-ended, whereas now everyone does some variation of the same thing (LLMs), concentrating effort and reducing exploration.
“If you look further into the past to like 2015 or 2016, there were like a thousand times fewer people doing AI back then. Yet the rate of progress was higher because people were exploring more directions.”
Scale maximalists believe that adding System 2 capabilities is relatively straightforward—you build a library of representations through pre-training, then add adaptive/test-time compute and an RL setup that rewards correct reasoning traces, distilling in-context learning back into the weights—and that this will be achieved within the next couple of years atop scaled models.
“Their perspective is that it's relatively straightforward to do that because you have this library of representations that you built up from pre-training.”
Dwarkesh's friends prompted Claude Opus to get 25% on the public ARC test by providing in-context examples that explain the reasoning of how one output was derived from another, expressing the JSON in a more tokenizer-amenable form, and using the code interpreter to write Python programs that produce the correct cell outputs.
“They were trying to prompt Claude Opus on this and they say they got 25% on the public ARC test.”
A genuine ARC solution trained only on core knowledge, while not necessarily AGI itself, would be a huge milestone because it represents the ability to synthesize a generalizing problem-solving program from just two or three examples—an entirely new paradigm where you program by showing the computer the desired output rather than typing up the program yourself.
“What it represents is the ability to synthesize a problem-solving program from just two or three examples. That alone is a new way to program.”
ARC gained little traction in the research community because benchmarks that gain traction are ones already fairly tractable—a group makes an initial breakthrough, attracting follow-up papers—whereas ARC is very hard for existing AI techniques and requires genuinely new ideas, so no initial breakthrough sparked the usual competitive dynamic.
“Benchmarks that gain traction in the research community are benchmarks that are already fairly tractable.”
When Scale AI replicated GSM8K with different questions, some models like Mistral were extremely overfit on the benchmark, but frontier models like Claude and GPT performed as well on the novel replicated benchmark as on the original public questions, suggesting frontier models are less overfit and would likely be relatively safe to test even if trained on the public ARC JSON.
“Scale recently did this with GSM8K. They basically replicated the benchmark, but with different questions. Some of the models actually were extremely overfit on the benchmark, like Mistral and so forth.”
The big labs tried their hand at ARC but, because they got bad results, did not publish anything—people only publish positive results—creating publication bias that obscures how poorly current techniques perform on the benchmark.
“The big labs also tried their hand on ARC, but because they got bad results they didn't publish anything. People only publish positive results.”
Chess at the highest level is all about memorization—chess grandmasters perform very well within the context of moves they have studied—offered as an example that even apparently sophisticated expert performance can be largely pattern matching and memorization rather than on-the-fly reasoning.
“That’s an excellent example because chess, at the highest level, is all about memorization, chess memorization.”
The January 2022 chain-of-thought paper revealed latent capabilities in LLMs that even an informed observer who thought they had priced in everything LLMs could do did not expect, which awakened Mike Knoop to the rate of AI progress and led him to leave his executive role to do AI research.
“It was January 2022 when the chain-of-thought paper came out. That really awoke me to the progress.”
The ARC Prize requires winners to put their solution or paper into the public domain, contrary to typical contests where people hold their alpha secret, because progress toward AGI requires that meaningful advances be shared and iterated on—if there's no sharing, there's no progress.
“in order to win the prize money, you put the solution or your paper out into the public domain.”
Today I have the pleasure to speak with François Chollet, an AI researcher at Google and creator of Keras, launching a prize in collaboration with Mike Knoop.
“Today I have the pleasure to speak with François Chollet, who is an AI researcher at Google and creator of Keras.”