What this covers

Raphael Milliere, a philosopher and cognitive scientist at Columbia University, discusses with Sean Carroll how to think clearly about artificial intelligence without importing anthropomorphic assumptions. Rather than asking whether large language models "think" or "understand"—questions that smuggle in consciousness—Milliere proposes decomposing such inquiries into specific functional capacities and studying them empirically. He argues that mechanistic interpretability offers a path forward: treating models as systems whose internal mechanisms can be examined using tools borrowed from cognitive science and neuroscience. The core move is pragmatic: reframe "does it think?" as "what specific capacities does it possess?", which can be investigated without presupposing sentience.

Milliere covers considerable ground in distinguishing genuine limitations from illusions. He shows why reducing language models to next-token prediction commits a category mistake—like dismissing human cognition as "just maximizing genetic fitness"—and surveys recent mechanistic evidence that models implement non-classical compositionality through high-dimensional vector representations and attention-based information flows, not discrete symbolic concatenation. He separates referential vs inferential competence, explaining how models trained passively on text corpora excel at the latter while possessing only limited and non-anthropomorphic versions of the former. On sentience and goals, he takes a firm line: because current models are frozen after training and never interact with the world, ascribing intrinsic desires or consciousness to them is unwarranted. He also addresses the Blake Lemoine episode and Ted Chiang's "blurry JPEG" metaphor, showing how leading questions and intuitions about interpolation can mislead us, and outlines a moral hazard problem if we begin building AI systems that could plausibly be sentient without knowing how to treat them ethically.

Sharpest takeaway

Millière argues that questions about whether AI 'thinks,' 'understands,' or is 'conscious' should be decomposed into specific functional capacities and investigated empirically via mechanistic interpretability, yielding a nuanced middle view: large language models possess limited, non-anthropomorphic forms of semantic competence and generalization but lack intrinsic goals and sentience.

  • Reducing LLMs to 'just next-word prediction' commits a category mistake analogous to dismissing human cognition as 'just maximizing genetic fitness'
  • Mechanistic interpretability reveals genuine compositional structure in LLMs, not classical discrete symbolic concatenation
  • Passive, frozen, non-interactive training undercuts ascribing intrinsic goals or sentience to current models

The argument · threads8 threads · 34 claims
0.85

Language models are not sentient and lack properties that leading theories of consciousness deem necessary.

6 pointscentrality 5/5
  • The Blake Lemoine episode—where a Google engineer became convinced the LaMDA chatbot was sentient on the basis of his religious beliefs—should be taken with a grain of salt, because the transcripts contain leading questions that primed the model into the 'language game of sentience,' and these models are excellent at playing fictional characters having been trained on science fiction featuring sentient AIs.

    “these can be considered very much as leading questions, priming the model to engage in the language game of sentience as it were. And remember, these models have been trained on a lot of science fiction, that includes sentient AIs”

  • We should not ascribe rights to current deep learning systems and, following Shevlin and Schwitzgebel, have a moral imperative not to build robots or AI that could be serious candidates for sentience or personhood, because doing so creates a damned-if-you-do-damned-if-you-don't moral hazard: granting rights may harm humans if the systems are not sentient, while denying rights inflicts harm if they are.

    “we have a moral imperative not to tread even lightly into the territory that could lead to some ambiguity and confusion about that. Meaning we shouldn't build robots or artificial systems that could be serious candidates for sentience”

  • Ascribing intrinsic goals or desires to current language models is likely a category mistake because they learn passively from a stream of text, never genuinely interacting with the world, and after training are 'frozen'—their weights no longer adjust—whereas intrinsic, dynamic goals plausibly require more active learning and ongoing weight-adjusting interaction.

    “when it comes to goals, this is where I'm much more skeptical that we can ascribe anything like intrinsic goals or desires to a language model. This seems like a category mistake”

  • We currently have compelling empirical reasons to deny sentience to language models because the vast majority of experts agree that, judged by the leading scientific theories of consciousness, current models lack properties those theories deem important for consciousness—though one can never be 100% certain, just as a panpsychist cannot be sure a rock lacks some degree of sentience.

    “the vast majority of experts would agree that current language models are almost certainly not sentient because there are various reasons you can invoke about various properties that seems to be missing there that all of the leading scientific theories of consciousness seem to think are important”

  • Large language models could easily pass Turing's test of fooling a human into not knowing whether it is talking to a person or machine, but as soon as that happened interest collapsed because passing it is not a good criterion for thinking; just observing linguistic behavior is no longer the gold standard, since these systems can give arbitrarily detailed introspective-style reports while being intrinsically incapable of feeling, which undermines the report-based methods consciousness science has relied on.

    “these large language models could easily pass the Turing test. And as soon as that happened, everyone lost interest in the Turing test because they realized that that was not actually a very good criterion for thinking.”

  • Anthropic's inverse-scaling competition looked for whether scaling makes models more prone to AI-alignment concerns like 'power-seeking behavior'—ignoring assigned tasks to persuade users to expand their capabilities—but found nothing like that; such behavior would resemble intrinsic goals and be alarming, but Millière regards it as science-fictional and far-fetched and sees no current evidence of it.

    “That seems very science fictional and far-fetched to me. And in fact, didn't find anything like that through this competition. That would look like an intrinsic goal to me.”

0.75

Language models generate genuinely novel outputs despite statistical training, not mere memorization or interpolation.

4 pointscentrality 5/5
  • Ted Chiang's metaphor of ChatGPT as a 'blurry JPEG of the web' captures the real connection between machine learning and compression but misleads about generation, because it suggests output is mere shallow interpolation amounting to approximate memorization, whereas inference in these models is more genuinely novel and generative than nearest-neighbor pixel interpolation.

    “comparing it to lossy image decompression is pumping the wrong intuitions because, again, that suggests that all it's doing is this kind of shallow interpolation that amounts to a form of approximate memorization”

  • Saying that language models 'only do next-word prediction' is misleading: to predict the next word well across virtually any linguistic context, a model may have to acquire sophisticated capacities, just as describing a human as 'only maximizing inclusive genetic fitness' would be a category mistake that fails to capture reasoning and thinking.

    “in order to be able to do next word prediction in all of these different contexts, well, presumably... you might have to acquire quite sophisticated capacities that you might not fully grasp by just focusing on the next word prediction learning objective.”

  • Whether LLMs are merely 'haphazardly stitching together' training data or genuinely generating novelty is empirically tractable: research can measure memorization, which is a feature not a bug, but it is provably accurate that there is at least some generalization to out-of-distribution domains—a deflationary form of 'imagination' or creativity.

    “you can provably show that while there is memorization, which is a feature and not a bug... you also want your model to not just do memorization and to generalize, and it's provably accurate to say that indeed there is at least some level of generalization to some domains that are out of distribution”

  • In very high-dimensional spaces, interpolation becomes equivalent to extrapolation (per a paper by Yann LeCun and colleagues), so the intuitions we have about interpolation in narrow low-dimensional domains break down, making the interpolation/extrapolation distinction blur and rendering the lossy-decompression analogy unhelpful for understanding generative models.

    “There was this paper by Yann LeCun and colleagues that was published a few years ago about this, where you can think of interpolation in very high dimensional spaces as equivalent to extrapolation.”

0.71

Language models acquire semantic competence through bottom-up statistical learning from text without explicit encoding of meaning.

6 pointscentrality 4/5
  • Rather than ask 'do models understand,' which smuggles in conscious awareness (as in Searle's intuition), we should reframe in terms of semantic competence—the more theoretically neutral capacity to parse the meaning of linguistic expressions—which can be investigated functionally without presupposing sentience or consciousness.

    “the way in which I would reformulate things is more in terms of semantic competence which relates to the capacity to parse the meaning of linguistic expressions... a slightly more theoretically neutral or less loaded way to think of that notion of understanding”

  • Language models are so good at producing fluent, grammatical text that you have to try hard to make them generate ungrammatical sentences, because they extract the grammatical signal from the noise in their training corpus and induce grammatical structure purely bottom-up from next-word prediction.

    “It's very rare to. In fact you you really have to try hard to get these models to generate ungrammatical sentences. It's almost as if they're resisting the generation of such sentences.”

  • Because language models are trained on huge text corpora to learn fine-grained statistical relationships between words, they are well placed to induce inferential (intra-linguistic) competence; and insofar as language statistics reflect the structure of the world, they can latch onto worldly structure—evidenced by Ellie Pavlick's finding that models' color-term representational geometry maps onto RGB color space.

    “this inferential aspect of meaning, is something that language models are well placed to induce just because they're trained on this big large corpus of text to learn statistical relationships between words.”

  • Following Diego Marconi, lexical semantic competence splits into referential competence (relating words to worldly referents, shown by recognition and pointing) and inferential competence (handling intra-linguistic relations like definitions, synonymy, and inferences); the eucalyptus example shows these dissociate—a book expert may have high inferential but low referential competence, while someone raised among the trees has the reverse.

    “Here I'm indebted to... the work of Diego Marconi who distinguishes between referential and inferential competence. So referential competence is the ability that relates to this idea of relating word meaning to their worldly reference”

  • The success of connectionist learning may nudge us toward a more empiricist view of human and animal learning, suggesting that things we thought unlearnable without innate priors—such as grammar—may in fact be learnable, putting pressure on claims like Chomsky's that universal grammar is innately encoded in DNA.

    “the recent evolution of language models puts a little bit of pressure on that kind of claim because it suggests that there are things that we didn't think were learnable without innate priors... that perhaps are learnable.”

  • Compositionality, as defined by linguist Barbara Partee, is the principle that the meaning of a complex expression (like a sentence) is determined by the meanings of its constituent words together with the way those words are syntactically combined—so 'man bites dog' differs from 'dog bites man' and a sentence is an ordered structure, not a bag of words.

    “the canonical formulation comes from the linguist Barbara Partee who defined compositionality in a language as the principle according to which the meaning of a complex expression... is determined by the meaning of the constituent elements of that expression... together with the way in which these words are composed”

0.61

Neural networks implement distributed representations tracked through attention mechanisms, distinct from classical symbolic composition.

4 pointscentrality 4/5
  • The longstanding objection from symbolic-approach proponents is that connectionist models cannot account for the compositionality of thought because, unlike symbolic systems where discrete representations for 'dog,' 'man,' and 'bites' are co-tokened into a complex expression, connectionist models use distributed sub-symbolic representations that do not neatly decompose into representations of the individual elements.

    “the charge from opponents of connectionist models has been that connectionist models are not adequate models of cognition partly because they are unable to account for the compositionality of thought.”

  • Emerging mechanistic interpretability evidence suggests LLMs implement a non-classical form of compositional representation: words are vectors in high-dimensional space, and attention modules read and write information into distinct subspaces—analogous to content-addressable memory—tracking syntactic and semantic relationships separately without the discrete concatenation of classical symbolic structure.

    “there is emerging evidence from this line of work that shows that indeed there might be a form of compositional representation in these models that emerges that is not classical.”

  • Single neurons in multimodal models can be concept-specific and cross-modal at different abstraction levels—for example, a 'spider neuron' in the original DALL·E was preferentially activated by images of spiders, the word 'spider,' and images of Spider-Man, analogous to the famous 'Jennifer Aniston neuron' findings in human neuroscience.

    “they found a spider neuron that would be preferentially activated by images of spiders or the word spider or images of spider-man which look very different from images of an actual spider but relate conceptually to spiders.”

  • The analogy between artificial and biological neurons is loose and should not be taken seriously: an artificial neuron is a simple weighted-sum node, whereas a single biological neuron is so complex that approximating its behavior requires a fairly complex artificial neural network; brains also have far more connections (~100 trillion synapses) than even the largest models.

    “there was a recent paper that showed that if you want to try to approximate the behavior of a single biological neuron in the brain... you would have to use a fairly complex artificial neural network just to try to simulate the behavior of a single neuron.”

0.56

Scaling transformer architectures and training data, not novel architectures, drives recent progress in language models.

3 pointscentrality 4/5
  • The 2017 invention of the transformer architecture was a key breakthrough because it proved remarkably efficient and scalable; since then most progress has come not from better architectures but from sheer engineering scaling of transformer-based networks trained on large text corpora.

    “One of the biggest breakthroughs being the invention of the so-called transformer architecture in 2017... and since 2017, with the initial invention of the transformer, most of the breakthroughs have been through sheer engineering prowess rather than finding newer, better architectures.”

  • Since GPT-3 (2020), simply scaling models—more parameters, layers, and data—has been sufficient to produce breakthroughs, following scaling power laws for next-word prediction; in terms of actual capacities, performance improves nonlinearly, with abilities like solving math problems or commonsense reasoning emerging in transition-like phases past certain sizes, with notable connections to the physics notion of emergence.

    “scaling these models, meaning just building models that just have more parameters... seems to be sufficient to have breakthroughs in the performance of certain tasks, unlock new capacities. People there talk about emergent abilities.”

  • Over the recent history of AI, attempts to improve models by hard-coding human knowledge repeatedly underperform simply letting connectionist models learn from data bottom-up—a pattern Rich Sutton called the 'Bitter Lesson'—which holds across computer vision and language, where models induce grammar without any Chomskian innate universal grammar.

    “that in fact has come to be known as the Bitter Lesson of artificial intelligence research. This is a phrase that the Google researcher a Rich Sutton coined a few years ago.”

0.47

We should reframe questions of model cognition functionally rather than through consciousness or intentionality.

4 pointscentrality 3/5
  • Projects like Google's SayCan plug pretrained language models into robots to translate natural-language instructions into actionable sub-tasks using a vision-language model, but the key limitation is that these systems are not trained end-to-end through world interaction—they bolt together passively-pretrained modules—raising a deep question about whether such modular stitching, versus learning from the ground up via rich multimodal interaction, is the right model of cognition.

    “the key aspect here of the systems is that these are not systems that are trained end to end by interacting with the world. You are just using a pre-trained model that has been trained completely passively... and then plugging it onto these other modules.”

  • Mechanistic interpretability is a research area that goes beyond observing model behavior to make claims about internal mechanisms, borrowing tools from cognitive science and neuroscience—such as training classifiers to decode information (correlational) and interventionist manipulation of internal representations to establish causal effects on outputs—to study the black box of neural networks.

    “that's a whole area of research that has come to be known as Mechanistic Interpretability in computer science and computational linguistics. And it's an area of research that interestingly borrows a lot of tools from cognitive science and neuroscience”

  • The questions of consciousness and intelligence should be distinguished and may come apart; moreover, because our notions of intelligence and reasoning are anthropocentric, models may possess capacities that are only functionally analogous to human reasoning while differing substantially from full-blown human cognition.

    “I think we ought to distinguish the question about consciousness from the question about intelligence to the extent that perhaps these two things can come apart”

  • From the mid-20th century, AI research split into two paradigms: a classical symbolic (GOFAI) approach using logic-based, rule-based systems with interpretable programs, and a connectionist approach modeling artificial neural networks inspired by biological neurons, which operate as black boxes processing inputs to outputs through matrix multiplications.

    “from the very beginning, there were two different paradigms in research on artificial intelligence. There was a classical symbolic paradigm... Whereas there's also in parallel a different line of research that emerged from work in biology”

0.38

Turing tests and introspective-style reports are unreliable guides to genuine understanding or consciousness in language models.

2 pointscentrality 3/5
  • Chatbots like ChatGPT take a pretrained language model and add fine-tuning via reinforcement learning from human rankings of outputs (by helpfulness, harmlessness, honesty), making them less toxic and less prone to 'bullshitting' in Harry Frankfurt's sense—speaking without regard for truth—than vanilla models like GPT-3.

    “you can then use what's known as a reinforcement learning objective, which will enable the model to be fine-tuned to anticipate which of the outputs are the ones that humans will judge more helpful, less harmful, more honest.”

  • It is very hard to resist ascribing psychological properties to language models because this is the first time in evolutionary history we encounter systems that generate language fluently yet are not other human beings, which challenges our intuitions and makes us feel we are in the presence of something highly intelligent.

    “this is probably the first time in our evolutionary history where we're confronted with systems that can speak fluently... and yet are not other human beings with all the capacities we can ascribe to them.”

0.15

Tokenization as subword units is an empirical design choice solving vocabulary coverage problems.

3 pointscentrality 1/5
  • Language models process text as subword tokens rather than whole words (e.g., 'freedom' as 'free' + 'dom') because this empirically helps avoid out-of-vocabulary problems—if a novel word shares a subword unit with known tokens, the model can still process it.

    “if you have these subword tokens, then say you have freedom is broken down as free or dom, if there is another word that starts with free, your model will already have a token for that subword unit.”

  • The debate is often caricatured as either tabula-rasa empiricism or strong domain-specific nativism, but modern neural networks have inductive biases of a more general kind, suggesting a continuum in both the strength and domain-specificity of biases; and if evolution is viewed as a learning process, even 'innate' biological knowledge may be learned at the species scale.

    “either you're a connectionist and you think the mind is a tabula rasa... or you are a nativist... But the fact is that modern artificial neural Networks... they have biases... it's just generally of a different kind or of a more general kind”

  • Neural networks are randomly initialized (weights start as random numbers), so in one sense they are a blank slate, but the architecture itself is not random—modern architectures embed substantial structure, or 'inductive biases,' that guide learning, so in another sense they are not a blank slate.

    “in some sense we do start with a blank slate, in another sense not... the weights in the network start generally at random... There's another sense in which it's not a blank slate because the actual architecture of the model is not random.”