What this covers

François Chollet joins Sean Carroll to argue that large language models are fundamentally incapable of intelligence in any meaningful sense. The episode centers on a provocative comparison: is ChatGPT or a three-year-old more intelligent? Chollet's answer hinges on a precise definition of intelligence—not skill at familiar tasks, but the ability to adapt to genuinely novel situations. He contends that LLMs are vast interpolative databases of vector functions trained through curve fitting, capable only of retrieving and blending patterns from their training data. True intelligence, by contrast, requires synthesizing entirely new solutions from minimal examples—something neither current LLMs nor any existing AI system can do.

The conversation moves through the machinery that creates this illusion of understanding. Chollet explains how LLMs as program databases operate within the convex hull of their training distribution, mimicking humans so effectively that people unconsciously grant them credit they do not deserve. He introduces ARC, a benchmark of never-before-seen puzzles where humans score around 85% and LLMs near zero, as empirical proof of this gap. The episode also touches on why skill and intelligence are often conflated—a student cramming memorized exams resembles an LLM exactly—and argues this conflation has corrupted education. Chollet discusses practical implications: LLMs excel at turning novices into mediocrity but struggle to improve experts, so they will likely have limited impact on scientific progress. He addresses existential risk, arguing that intelligence itself does not entail danger; danger requires deliberately engineered autonomy. Throughout, he remains skeptical that any current research direction leads toward AGI, though its arrival remains unpredictable.

Sharpest takeaway

Chollet argues that LLMs are not intelligent in any meaningful sense—they are vast memorized databases of interpolative programs that can mimic understanding by fetching patterns, whereas true intelligence is the ability to synthesize novel solutions to genuinely new problems, which no current AI can do.

  • LLMs are curve-fitting systems limited to interpolation within their training distribution and cannot generate genuine novelty.
  • True intelligence is the efficiency of converting past experience into the ability to handle novel situations, not skill at familiar tasks.
  • Benchmarks like ARC, which use never-before-seen puzzles, show humans succeed easily (~85%) while LLMs score near zero, exposing the gap between memorization and intelligence.

The argument · threads10 threads · 40 claims
0.85

LLMs are interpolative databases of semantic programs, not reasoning systems.

9 pointscentrality 5/5
  • The core idea of deep learning and LLMs is that relationships between things can be represented as literal distances in a vector space; tokens are mapped to points whose geometric proximity encodes semantic similarity, similar in spirit to Hebbian learning where neurons that fire together wire together.

    “The big idea behind LLMs and behind deep learning in general is that the relationship between things can be described in terms of distance between things like a literal distance.”

  • Through training on massive data in high-dimensional space, LLMs build a smooth differentiable manifold that, due to compression pressure, encodes millions of reusable semantic 'programs'—vector functions mapping subsets of vector space to others—so the best way to think of an LLM is as a vast interpolative database of programs rather than a reasoning system.

    “the most effective way of thinking about LLMs is that they are big stores of programs, millions of programs... they are more like vector functions, right? And that means that you can actually interpret between different programs.”

  • When prompted, an LLM does not understand or simulate the query as a human would; it fetches a memorized program or interpolates between memorized programs and reapplies it to the input, which works well for familiar patterns seen many times in training but fails on unfamiliar inputs—and crucially the LLM has no way to tell the difference, since it does the exact same thing whether correct or completely off.

    “any time you find a query where you're getting the right answer, it is most likely possible to modify the query a little bit in a way that would be totally transparent to a human... but it will cause the LLM to start failing.”

  • Inside an LLM you find not a symbolic model of the world but a model of semantic space that partially overlaps with a world model yet is fundamentally different and far less generalizable; a symbolic model that enables simulation knows the rules generating situations and can imagine entirely novel ones, whereas an LLM only knows specific situations and can interpolate across them without moving beyond those interpolations.

    “In an LLM, you will not find a symbolic model of the world but you will find a model of world space, a model of semantic space... any sort of symbolic model that enables simulation is gonna be able to generalize much further away from what it has seen before, because it does not just know about specific situations. It knows about the rules that generated this situation.”

  • Interpreting how LLMs work is an active, tractable research area making progress, and every new result—including a recent Anthropic paper—shows LLMs are pattern-matching engines and interpolative databases of programs rather than reasoning systems; the difference between intelligence and a program database is like the difference between a programmer and GitHub, since no amount of stored programs suddenly yields the ability to synthesize new ones.

    “every time we get new results, they are along the line of showing that LLMs are actually just pattern matching engines... the big difference between intelligence and a program database is the program database is like GitHub. Intelligence is like the programmer.”

  • Because LLM transformations are vector programs, you can average or interpolate between them—asking it to talk like a pirate and like Shakespeare simultaneously yields a Shakespearean pirate—which is something you cannot do with an explicit logic program, giving the impression of creativity through interpolation within the convex hull of training representations.

    “you could ask an LLM to talk like a pirate, but you could also ask it to talk like Shakespeare... you can actually merge them. You can average them, you can interpolate it between them. And that means you can start talking like a Shakespearean pirate”

  • LLMs are all about mimicking humans, so they are very good at hacking your theory of mind because humans have a bias toward interpreting anything that superficially acts like them as being like them—which is why a short text snippet from an LLM is hard to distinguish from a real person's and tends to be given more credit than it deserves.

    “LLMs are all about mimicking humans, so they're very good at hacking your theory of mind because you have this bias towards interpreting as being like you anything that superficially act like you.”

  • The claim that LLMs do in-context learning is a misconception; LLMs do no learning—given a slightly novel but similar problem they fetch or interpolate the correct rule set rather than an answer, which is one level higher than answer-fetching but still memory retrieval, not genuine learning that adjusts the underlying curve.

    “lots of people talk about oh like LLMs can do in-context learning, but that's actually a total misconception. LLMs do no learning. What they can do is that given a new problem that is slightly novel but still very similar to something they've seen before, they can fetch the correct program”

  • Word2vec illustrates semantically meaningful vector functions in miniature: words are points in vector space such that adding a consistent vector transforms words in a consistent way—king minus man plus woman equals queen—demonstrating that meaningful operations like pluralization or gender shift can be encoded as vector addition.

    “you have words represented as points in the vector space, and you can actually add a certain vector to any point to get a new point, which is a new word... For instance, making a word plural or going from a male word to a female word”

0.72

LLM brittleness and sensitivity to input phrasing reveal dependence on memorization over understanding.

5 pointscentrality 5/5
  • Because LLMs are big differentiable parametric curves fit to a data distribution, you cannot input symbolic rules valid for whole categories of problems; you can only supply individual data points the model interpolates between, so fixing errors requires labor-intensive pointwise patches that address one query at a time rather than entire categories of issues.

    “It is virtually impossible to fix a general category of issues at once. And the reason why is because these models, they're basically big curves, they're big differential parametric curves that are fit to a data distribution. And so you cannot really input into them symbolic programs for instance.”

  • LLMs are very sensitive to phrasing and brittle, which gave rise to prompt engineering; for virtually any query that yields a correct answer, you can rephrase it in a way that is completely transparent to a human yet causes the LLM to fail, simply by making the query slightly more unfamiliar relative to its training data.

    “any time you find a query where you're getting the right answer, it is most likely possible to modify the query a little bit in a way that would be totally transparent to a human... but it will cause the LLM to start failing.”

  • Besides unfamiliarity, LLMs fail the opposite way: when over-familiar with a pattern they cannot deviate from it, as shown by the steel-versus-feathers weight puzzle—early ChatGPT-4 answered 'they weigh the same' even when asked about one kilogram of steel versus two kilograms of feathers, because it fetched the memorized pattern instead of reading the actual numbers.

    “If you asked it what's heavier, like one kilogram of steel or two kilograms of feathers, it would be, "Oh, they weigh the same. I know the answer. They weigh the same." So it's not actually trying to read and understand the query. It's just fetching the pattern and reapplying it.”

  • When you tell an LLM its answer is wrong and it corrects itself, it is not genuinely reconsidering but applying a learned modification function from incorrect-to-fixed solutions; in the absence of human feedback there is no reason the modified program is more correct than the initial one, which is why it cannot do this preemptively—only the human can tell which answer is right.

    “in the absence of human feedback, there is no particular reason for the modified program to be more or less correct than the initial program. Like only the human can tell.”

  • When showing skill at familiar tasks you can always trade off intelligence for memorization; just as a student who crams thousands of past exams can score high by fetching memorized patterns without understanding the concepts, LLMs—being superhuman memorization machines—produce the appearance of skill and understanding with no actual understanding, which is why the school system's reliance on memorizable tests fails to distinguish the two.

    “when it comes to showing skill at something at something familiar then you can always trade off intelligence for memorization.”

0.68

Intelligence is the ability to learn novel tasks from few examples; LLMs lack this core capacity.

4 pointscentrality 5/5
  • Intelligence is not skill at many tasks but the ability to pick up new skills and adapt to situations never seen before; what makes intelligence general is not that it scales to many X-Y-Z tasks but that it can scale to an arbitrary new task you teach it, so a static chess engine or a frozen curve-fitted model has zero intelligence no matter how skillful it is within its distribution.

    “intelligence according to me, is the ability to pick up new skills, to adapt to new situations, to things you've not seen before.”

  • Humans can pick up a new task from a couple of demonstration examples, vastly more data-efficiently than LLMs; Chollet's three-year-old, having assembled fewer than a thousand Lego bricks and seen only a handful of Lego cars, can imagine and build entirely novel complex constructions—something no AI system today can do, demonstrating that a three-year-old is far more intelligent than any LLM.

    “he's assembled a total of probably fewer than 1000 Lego bricks in his entire life. But no it can actually create new things really complex new things. So LLMs can definitely not do that.”

  • General intelligence is the ability to synthesize on the fly a new correct program matching a novel problem from just a few examples, rather than fetching or interpolating memorized programs; if a system could do this efficiently it would have AGI and would also be able to solve ARC, whereas LLMs fail precisely because all they can do is fetch.

    “It can only give you things that are simple interpolations of programs, code snippets that it has seen before... you're not gonna get an LLM which has no novel problem solving ability. You're not gonna get it to invent AGI.”

  • When a programmer writes an exact static program describing a task so precisely that no uncertainty or novelty remains, the resulting program—though people call it an AI—contains no intelligence; the only intelligence in the picture is the mind of the programmer who understood and modeled the problem.

    “The only intelligence in the picture is the intelligence of the programmer that understands the task, understands the problem, models it in their head. And then writes down exact instructions... The intelligence here is the mind of the programmer that developed that program.”

0.55

LLMs should augment human judgment as stepping stones, never replace decision-making or autonomous action.

4 pointscentrality 4/5
  • Foundation models are good at giving directionally accurate answers that point in the right direction but are very bad at giving exactly correct answers, especially for quantitative problems, so they should be used as a stepping stone toward a solution that you then refine yourself or with a symbolic search system, never blindly trusted.

    “they're very good at giving you answers that are directionally accurate, that are a step in the right direction but they're extremely valid giving you exactly correct answers. So you should pretty much never ask a foundation model to give, especially if it's a quantitative problem by the way, to give you an exact answer and then just blindly use that answer.”

  • Many people try to use generative AI for things it is not suited for; the advice is to never delegate decision-making to an LLM—do not let it generate your emails or make your decisions—but rather use it as a shortcut toward the general area you are looking for, such as helping you write faster or fixing typos.

    “do not try to delegate any sort of decision-making to the LLM. The LLM is there to give you a shortcut towards the general area that you're looking for. Do not delegate your decision.”

  • Intelligence does not imply autonomy—the ability to perceive and act in the world without human mediation—and no machine today has autonomy, if only because none can recharge and maintain itself in perpetuity without a human-supplied power source, so creating real danger requires deliberately engineering autonomous sensing and acting.

    “There is no machine out there today that is unmediated from humans. If only because they need a power supply. There's no machine out there that can just recharge itself and maintain itself in perpetuity.”

  • The fear that AGI will end humanity rests on misconceptions: intelligence is just a conversion ratio from information to the ability to act in novel situations—a pathfinding ability—that does not entail goals or values; to be dangerous like SkyNet a system must additionally be given autonomous sensing, autonomous acting, its own value system, and autonomous goal-setting with no human supervision, so danger must be deliberately engineered and the real risk is humans misusing the tool, not the tool competing with humanity.

    “intelligence is pretty much just a conversion ratio between the information you have to the ability to operate in novel situations in the future... this ability to sort of like chart a path through situation space does not entail that the system should have goals of its own”

0.51

Only open-ended search over unconstrained spaces produces genuine invention; LLMs cannot invent.

3 pointscentrality 4/5
  • Real invention and creativity by an algorithm has historically occurred only when an open-ended search process operates over a relatively unconstrained search space; LLMs cannot invent anything because they are limited to interpolations, whereas genetic algorithms with the right search space and fitness function can invent entirely novel systems no human could anticipate—such as the famously novel antenna design.

    “when we saw real invention, real creativity initiated by an algorithm, it's been in cases where you had a very open-ended search process operating over a relatively unconstrained search space.”

  • An LLM cannot bootstrap a smarter AI because it is constrained by its training data to interpolating known code snippets; even an average senior software engineer—a tremendously capable novel problem-solver—cannot invent AGI, so an LLM with no novel problem-solving ability certainly cannot, and cannot even solve a trivial ARC problem that a four-year-old can.

    “It can only give you things that are simple interpolations of programs, code snippets that it has seen before... you're not gonna get an LLM which has no novel problem solving ability. You're not gonna get it to invent AGI.”

  • Neural architecture search has never produced a popular architecture despite enormous compute dedicated to it; every important architecture—Transformers, Mamba, xLSTM—was invented by humans, because humans are really good at inventing and AI is not idea-constrained today, making it pointless to outsource ideation.

    “There's literally nothing out there that's popular that was developed by an algorithm, despite tremendous amounts of compute dedicated to this idea. And everything out there, like Transformers... or even the more novel and recent architectures like Mamba or xLSTM and so on. All of these were invented by humans”

0.51

ARC benchmarks reveal that LLMs fail fundamentally where human cognition excels on novel reasoning.

3 pointscentrality 4/5
  • Chollet is relaunching the ARC competition on Kaggle after four years with over one million dollars in prizes, aiming to reach roughly human-level (85%) performance, explicitly to incentivize people to invent new approaches beyond LLMs—which he characterizes as an off-ramp rather than a path to AGI.

    “We're gonna have over $1 million in prices. And the goal is to solve ARC to pretty much human levels. So something like 85%.”

  • ARC (Abstraction and Reasoning Corpus for AGI), released in 2019, is a benchmark of unique, original puzzles never seen before and partly kept private, designed so that memorization cannot help; humans solve them easily while deep learning and LLMs score very poorly—GPT-3 scored zero in the 2020 Kaggle competition—because each puzzle is genuinely novel and LLMs have essentially no ability to adapt to such novelty.

    “you come up with puzzles that are all unique, all original, never seen before, not similar to anything you would find on the internet... deep learning methods and LLMs in particular have scored very poorly on ARC.”

  • To distinguish genuine understanding from memorization, you must give students or models a novel problem never asked before that requires first-principles reasoning; students who merely crammed and the LLM will both score zero, while the smart student who understood the concepts but didn't memorize will excel—even though testing this way requires the labor of inventing fresh problems each time.

    “instead of just giving your students or the LLM a problem that's derivative... you come up with something novel... you actually need to think from first principles. You actually need to understand the underlying physics concepts”

0.34

Manual human annotation labor, not mere next-word prediction, makes foundation models useful.

2 pointscentrality 3/5
  • Every company training foundation models employs several thousand people full-time to create and rate training data by correcting answers to queries the model handles poorly; this manual annotation is what actually makes the models useful, beyond mere next-word prediction across internet text.

    “every company out there that's training this foundation models is employing typically several thousands of people just doing this full time. And this is, by the way, this is very much what makes these models useful.”

  • Curve fitting via gradient descent is a form of learning with non-zero but very low intelligence because it converts data into solutions at an extremely poor, data-inefficient conversion ratio; a truly intelligent system would let you fix an error by explaining why an answer was wrong and have it automatically patch the entire underlying category of issues, rather than requiring pointwise fixes.

    “a curve trend with gradient descent has non-zero intelligence... Because it turns data into solutions at some rate according to some sort of like conversion ratio, which is not a very good conversion ratio, by the way. It's extremely data inefficient. But it has very, very low intelligence.”

0.34

Democratizing deep learning tools empowers all developers to solve their own problems independently.

6 pointscentrality 2/5
  • It would be a sad world if only a fixed set of companies trained models and gave them to others as mere consumers; deep learning should be a tool in the hands of everyone so every software developer can tackle their own problems, which motivated making Keras as accessible and approachable as possible.

    “I think it would be a sad world, if there were only a fixed set of companies training models, and just giving those models to other people to be consumers of those models. I think we want this technology to be a tool in the hands of everyone.”

  • The transformer architecture, developed in late 2016 and made public in 2017, replaced recurrent networks like LSTMs (which date to the early 1990s) for sequence processing and works so well that nearly anything—images, video—can be treated as sequence data, enabling the rise of internet-scale generative AI.

    “everyone is pretty much using the transformer architecture at this time which was developed in late 2016 and got public in 2017. And it works really well, and it works for sequence data but pretty much anything can be treated as sequence data.”

  • A business making extensive use of LLMs should fine-tune its own models on its own data because this builds a private repository of programs and lets the model fit the right program from your data rather than merely fetch a public one, giving a competitive advantage over reusing the same public program database as everyone else.

    “if as a business, you want to make extensive use of LLMs, you should be finding your own LLMs because this gives you an advantage instead of just reusing the same program database as everyone... You are starting to develop your own repository of private programs”

  • There is no technology today that is on the path to AGI—nothing that simply scales into intelligence—so AGI's arrival cannot be predicted because it requires inventing something genuinely new; this does not mean AGI is far away, only that its timing is unpredictable, and it could be invented next year or much later.

    “we have no technology today that is on the path to AGI. There is nothing that if you just scale it, it gives you intelligence... you cannot predict when it will arrive because you need to invent something new.”

  • Empirically, LLMs are very good at turning people with no skill into people capable of an average mediocre outcome, but extremely bad at helping someone who is already excellent get better; since scientific progress depends on the top papers rather than more mediocre ones, LLMs in isolation will not have much impact on science.

    “LLMs are very good at turning people who have no skill into people who are capable of an average mediocre outcome... They are extremely bad at helping someone who's already extremely good getting better.”

  • Although intelligence was produced by evolution and could in theory be re-derived by a search algorithm, brute-forcing AGI via genetic search is a bad idea in practice: the required scale is excessive, humans already are general intelligence and are extremely effective at predicting which ideas to try, so outsourcing cheap ideation to a bad algorithm while leaving expensive evaluation to be done anyway is ineffective—and the search may not even contain the correct solution.

    “general intelligence gives you an extremely effective ability to predict what next idea should be tried. If you try to delegate this sort of ideation bit to an algorithm, you are wasting resources”

0.31

LLMs have plateaued because training data is exhausted; scaling alone cannot overcome architectural limits.

1 pointcentrality 3/5
  • LLMs entered a plateau since last year because we have run out of novel training data; since the curve is just a representation of the training dataset, scaling compute alone becomes useless, so improvements now come from curating higher-quality data rather than more data, and GPT-5 (likely released end of year) will be a disappointment not meaningfully better than GPT-4.

    “the reason why LLMs have entered the plateau since last year. It's because we've been running out of data... If you've run out of data, then how do you improve the model?”

0.24

Deep learning's current dominance obscures older machine learning methods and neural network history.

3 pointscentrality 2/5
  • Machine learning did not begin its popularity with connectionist methods: after early neural networks came SVMs, then random forests (popular ~2000-2010), then gradient-boosted trees—none neural-net or curve-fitting based—before the deep learning rebirth around 2011-2012 with GPU-trained convolutional networks, culminating in the 2012 ImageNet breakthrough.

    “one of the first big breakthroughs of machining were SVMs... After that, random forests got very popular... Then, Gradient-boosted trees got also very popular... And after that, you had the great rebirth of neural networks with the rise of deep learning.”

  • Neural network research is largely grounded in ideas from the 1980s and 1990s, yet most people doing deep learning today have very little knowledge of anything that came before about 2015.

    “I feel like, most people doing Deep Learning today have actually very little knowledge of anything that came before like 2015 to be honest.”

  • The fundamental dichotomy in AI is not symbolic versus connectionist, but between programmers hard-coding a symbolic model of a task versus systems that learn from data how to perform a task; symbolic approaches are more tractable with limited compute, while learning approaches require more computational resources.

    “The big dichotomy here is actually between, having programmers hard code, model, symbolic program of the task that they wanna do versus having a system that can actually learn from data how to perform the task.”