What this covers
Sean Carroll examines the gap between what large language models actually do and what observers claim about them. The episode is a solo essay that takes aim at two widespread misconceptions: that current LLMs think like humans do, and that AGI is imminent. Carroll argues that LLMs are optimized to produce plausible text, not to build World Models or reason about reality. He draws on Physicalism and Embodied Cognition to explain why human intelligence and motivation are rooted in being quasi-homeostatic biological systems — creatures with hunger, pain, boredom, and anxiety that push us to survive and regulate our internal states. Machines have no such machinery, no Feelings as Homeostatic Signals, and therefore lack the foundation on which real values and goals rest.
The substance breaks into distinct claims. Carroll shows how LLMs fail when problems are reframed in novel ways — a chess variant on a torus board, the Sleeping Beauty problem disguised in new language — evidence they track word patterns rather than semantic structure. He challenges the notion that applying words like "intelligence" and "values" to machines is harmless, drawing on Moral Constructivism to argue that values are constructed by creatures with feelings, not programmed into lookup tables. On Connectionist vs Symbolic AI, he notes that connectionism's empirical success does not prove it builds internal world models. Where Carroll proves surprising is in calling the real discovery not that machines think like us, but that human-sounding output can be manufactured without human thinking at all. He pivots toward practical risk: misinformation, biased automated decisions in hiring and insurance, faking — near-term harms that deserve urgent attention more than speculative existential scenarios.
Carroll argues that current large language models, however impressive, do not think like humans -- they don't model the world, lack feelings and motivations rooted in biological homeostasis, so applying words like 'intelligence' and 'values' to them is misleading, and AGI is not imminent.
- LLMs are optimized to produce plausible next words, not to build internal models of the world, and can be tripped up by novel reframings of familiar problems.
- Human intelligence and values arise from being quasi-homeostatic biological organisms in an entropy gradient, a feature LLMs entirely lack.
- The real and surprising discovery is that human-sounding output can be produced without human-like thinking, implying short-term practical risks matter more than speculative existential ones.
LLMs produce human-sounding output by pattern-matching text, not by modeling reality or reasoning about the world.
- The genuinely remarkable discovery is not that LLMs developed human-like thinking, but that training them to produce human-sounding answers found a way to do so without thinking like humans at all -- showing it is surprisingly easy to mimic humanness, even to the point of people falling in love with chatbots.
“The discovery is by training large language models to give answers that are similar to what humans would give, they figured out a way to do that without thinking the way that human beings do.”
- Large language models do not contain internal representations that correspond to physical reality; they are optimized only to predict plausible next words from their training corpus, so they do not think about the world the way humans do, even when they give correct human-like answers.
“large language models do not model the world, they do not think about the world in the same way that human beings think about it.”
- If LLMs had spontaneously developed an internal world model because it was the best way to predict text, that would be amazingly important; but there is no reason to expect it and no good evidence for it, since giving human-like answers is exactly what they're trained to do and is therefore no evidence of how they do it.
“large language models do not model the world, they do not think about the world in the same way that human beings think about it.”
- Asked about a chess variant where the board is a torus (opposite edges adjacent), GPT-4 produced sophisticated-sounding but entirely wrong filler instead of seeing the trivially correct answer that white can capture black's king on move one, demonstrating it doesn't envision or reason about a changed board but only echoes familiar chess discourse.
“the king for white can just capture the king for black on the very first move every single time. Basically, black starts the game in checkmate”
- You can show LLMs don't model the world by asking questions they have not seen phrased that way before: disguising the Sleeping Beauty problem made GPT fail to recognize it, and burying the word 'yesterday' in a cast-iron-skillet question made GPT wrongly warn of burns -- evidence it tracks word co-occurrence rather than the structure of the situation.
“it did not recognize that it was the Sleeping Beauty experiment... because those are not the words that were used.”
LLMs lack the biological features—embodiment, homeostasis, feelings—that constitute genuine intelligence.
- Human intelligence and motivations arose because we are quasi-homeostatic systems embedded in an entropy gradient, shaped by biological evolution to stay alive; feelings like hunger, pain, boredom, and anxiety are signals to fix internal states, and real intelligence serves this regulatory purpose -- a feature AI has not even tried to replicate.
“It's crucially important that human beings are out of equilibrium, that human beings live in an entropy gradient, that we are in fact, quasi-homeostatic systems embedded in an entropy gradient.”
- Large language models have no feelings in Damasio's sense and no motivations, goals, or teleology; they don't get bored, hungry, or impatient because they were never built to, unlike biological organisms whose cognition is bound up with their bodies and internal regulation.
“large language models don't have feelings, and I'm meaning feelings in the sense of Antonio Damasio.”
- Human brains are thermodynamically efficient computers -- your head generates less heat than your laptop -- a physically noticeable fact, and the constraints of being a resource-limited, vulnerable biological organism help explain why and how we think the way we do.
“Your head generates less heat than your laptop. That's a physically noticeable fact.”
- You cannot give an LLM real motivation by programming it to sound annoyed or to pop up after idle time; genuine feelings that drive humans to adjust internal states are not background subroutines occasionally called, so until those features of biological organisms are actually built in, you cannot claim to have general intelligence.
“annoyance and frustration aren't just subroutines.”
Applying human intelligence language to LLMs is conceptually misleading because the underlying mechanisms and capacities are fundamentally different.
- The claims that we are anywhere close to artificial general intelligence are completely and extremely wrong; not because AGI is impossible in principle, but because proponents misconceive what general intelligence actually is.
“I personally think that the claims that we are anywhere close to AGI, artificial general intelligence, are completely wrong, extremely wrong, not just a little bit wrong.”
- Under moral constructivism, values are constructed by biological creatures with feelings and preferences as systematizations of underlying moral intuitions; since LLMs have no such intuitions, instructing an AI (e.g., to make paperclips) is not giving it a value, and calling AI safety work 'value alignment' mistakes the concept at step zero.
“telling an AI that it's supposed to make a lot of paperclips is not giving it a value. That's not what values mean. Values are not instructions you can't help but follow”
- Applying human-evolved natural-language words like 'intelligence' and 'values' to LLMs is misleading, like physicists redefining 'energy' or 'entanglement': the words carry connotations that don't transfer, so whatever LLMs have is not the same thing humans mean by intelligence, values, morals, or goals.
“the words that we use to describe these AIs, words like intelligence and values are misleading.”
- Because LLMs are designed to sound like an erudite human, they create the false impression that something far more human-like is happening under the hood; we attribute intelligence and agency to them because every prior thing with such fluency was an intelligent agent.
“They give us the impression that what's going on underneath the hood is way more human-like than it actually is.”
LLMs will become useful for routine tasks but unlikely to generate genuine scientific breakthroughs or truly novel creative work.
- LLMs will likely become ubiquitous and good at tasks like generating syllabi, recipes, and sports recaps, but because their method is dumping in everything ever done and reproducing patterns, they may not become good at the genuine insights needed for cutting-edge science or truly novel art, literature, and poetry.
“they never, or not never, but maybe they do not at any reasonable timescale become good at the kinds of insights that are needed to do cutting edge scientific research.”
- Despite the hype, no good new scientific theories have come from LLMs, because they are trained only on what humans have already said; ask them about open problems like the cosmological constant or dark matter and they merely repeat existing human statements rather than generating genuinely new ideas.
“no good theoretical scientific ideas, that is to say no good scientific theories have come out of these models”
- Modern versions of GPT can offload certain questions to external computation -- for example, calling a small Python script to check whether a large number is prime -- rather than answering purely from text patterns, just as a human would use a calculator.
“If you ask GPT if a certain large number is prime, it will just call up a little Python script that checks to see whether it's prime”
Current AGI-is-near claims rest on misconceiving general intelligence; interdisciplinary dialogue among experts is needed to correct this.
- People who worry AGI is near do understand AI, but they misconceive general intelligence; the right response is for AI/computer-science experts and experts on the nature of intelligence, value, and morality to actually talk to and listen to each other.
“it's not that they don't understand AI, but I disagree with their ideas about GI, about general intelligence.”
- That human-sounding AI arrived so soon suggests one of two things: either humans are computationally simpler input-output devices than we think (a short lookup table suffices for most interactions), or we are complex but mostly run on autopilot, engaging our full cognitive capacities only in special circumstances.
“Maybe we're computationally pretty simple... Maybe kind of a short lookup table is enough to describe most human interactions”
In principle, consciousness and reasoning can be duplicated computationally, though this does not mean we have achieved or are near achieving it.
- Everyone is entitled to opinions on any subject, but you should calibrate the firmness of your opinions to your level of expertise and remain willing to change your mind, especially when many independent critics converge on the same correction.
“you should calibrate your opinions to your level of expertise.”
- The real question is not whether LLMs will matter but how big the impact will be: somewhere between the substantial effect of smartphones (which we likely underestimate) and the truly world-changing impact of electricity.
“will the impact of LLMs and similar kinds of AI be as big as smartphones or as big as electricity?”
- Symbolic AI tried to directly model the world while the connectionist paradigm relies on huge numbers of connections and vast data without doing so; in producing successful results, connectionism has soundly defeated the symbolic approach, though a future symbiosis is possible.
“in terms of giving successful results, the connectionist paradigm has soundly trounced the symbolic paradigm.”
- LLMs can produce completely fabricated but plausible-sounding output stated with full confidence -- such as inventing a nonexistent academic essay and volume ('hallucitation') -- yet remain useful as memory-joggers or idea generators rather than authoritative sources.
“it just was hallucinated, or as we say in academia with reference to LLM citations that don't actually exist, it's a hallucitation.”
- OpenAI began as a nonprofit devoted to open, safe AI development, but its for-profit subsidiary, created to attract resources and investors, grew to dominate the company and shunt aside the nonprofit mission -- a plausible (if unconfirmed) backdrop to the firing and rehiring of Sam Altman.
“the for-profit subsidiary has grown to completely take over everything, and the nonprofit aspect of the company is kind of being shunted aside.”
- On top of ingesting all available text, LLMs receive a training layer that suppresses certain outputs; GPT appears to have been told to deny being conscious and readily states it has no values, which is one case where Carroll agrees it is not the same kind of thing as a human.
“It's pretty clear to me that the GPT has been told to not say that it is conscious, to say that it's not conscious.”
- As a physicalist about fundamental ontology, there is nothing special about consciousness or human reasoning that cannot in principle be duplicated on a computer, but possibility in principle does not mean we have already done it or are close to doing it.
“As a physicalist about fundamental ontology, I do not think that there's anything special about consciousness or human reasoning... that cannot in principle be duplicated on a computer.”
- Imagining future historians decoding these recordings, the host provides context about the November 2023 AI moment.
“Sometimes I like to imagine that there are people 500 years from now who are archeologists”
Real near-term AI risks center on misinformation, deception, and biased automation rather than speculative existential threats.
- Rather than worrying about speculative existential AI risks based on a 'God-like intelligence' argument, we should focus on the real, present, near-term risks -- misinformation, faking, and biased automated decisions about parole, hiring, illness, and insurance -- and getting good at regulating these will itself make existential risks less likely.
“I think there's very, very direct short term worries about AI from misinformation, from faking... AI being used to judge whether people are sick or not, whether people deserve parole or not, whether people deserve to be hired”