235 | Andy Clark on the Extended and Predictive Mind
What this covers
Andy Clark, a philosopher and cognitive scientist, sits down with Sean Carroll to defend a counterintuitive picture of the mind: neither a brain-bound computer nor a passive receiver of sensory data, but an active prediction machine distributed across body, world, and tools. The core claim is that the brain's fundamental operation is generating expectations about incoming sensory signals and constantly minimizing prediction error—the gap between what it predicts and what actually arrives. From this single principle, Clark unpacks how perception, action, and even our reliance on notebooks and smartphones all serve the same underlying purpose: keeping predictions aligned with reality.
The conversation traces this framework through domains where it yields unexpected clarity. Stage magic works because magicians exploit how the predictive processing system allocates attention only to where it expects informative signals, leaving dramatic misdirection invisible. Pain reprocessing therapy succeeds by reframing pain not as injury signal but as prediction error from the nervous system itself, allowing patients to move more freely as their brain updates its model. Optical illusions like the hollow-face effect reveal how strong predictions can override sensory evidence—though notably, people on the autism spectrum prove slightly less susceptible, suggesting their exteroception is weighted differently. The Extended Mind thesis emerges too: gesturing during problem-solving, scribbling while thinking, and reaching for a smartphone all offload prediction work into the physical world, making them genuine parts of cognition, not mere aids to it.
Clark addresses complications head-on. He clarifies that "perception is controlled hallucination" inverts the relationship: hallucination is better understood as uncontrolled perception, when predictions run untethered from sensory constraint. He notes that evolution seems to have favored the metabolic gamble of maintaining always-running generative models over cheaper, more reactive systems. And he concedes what every account of mind has learned—that his framework, too, will eventually prove insufficient, leaving the question what stepping stone it becomes.
Clark argues the mind is not a passive computer inside the skull but an extended, embodied system that works by constantly predicting its own sensory inputs and minimizing prediction error, so perception, action, and even external tools are all manifestations of one predictive process.
- Cognition can extend beyond the brain into reliably accessible external resources like notebooks and smartphones
- The brain's canonical operation is guessing sensory signals and minimizing prediction errors, where perception updates predictions and action changes the world to fit them
- This predictive framing explains illusions, attention, curiosity, placebos, and pain as features of error minimization
The brain constantly predicts sensory flow and processes only prediction errors, making perception fundamentally generative rather than receptive.
- Predictive perception works like motion-compressed video and JPEGs: the brain predicts the current sensory flow and only the residual prediction errors, which carry whatever sensory information is currently unexplained, need further processing, so if you predict correctly using what you know about the world you need very little bandwidth to refine the picture.
“it's exactly like JPEGs and motion-compressed video where the idea is, if I've already transmitted the information about this frame of the video, then in order to know what the next frame is, all I need to do is react to whatever the errors would be”
- Well-predicted sensory inputs cause less neuronal activity than other inputs, which is a signature that led to predictive coding; attention reverses this dampening effect, so giving new evidence its proper due requires deliberate effort to attend against the grain of established expectations.
“well-predicted sensory inputs cause less neuronal activity than other sensory inputs”
- The canonical computation of the brain is attempting to predict the current flow of sensory information; at the basic time scale it is predicting the present, acting as a guessing machine that combines prior experience with current sensory evidence to estimate the true state of the world, which lets us use the past to make better sense of present sensory signals.
“this is a canonical operation of the brain... it's basically saying this is a canonical operation of the brain”
- Perception and action are two ways of getting rid of prediction error using the same basic neuronal operations: perception minimizes error by finding a better prediction or retrieving an old model of the world to fit the sensory flow, while action minimizes error by changing the world to fit the prediction.
“you have these two ways to get rid of prediction error, you find a better prediction or you change the world to fit the prediction you got. One is perception, the other is action.”
- The predictive architecture trades the metabolic cost of maintaining an always-running generative world model against the moment-by-moment cost of processing incoming information; this upfront cost buys faster, more efficient, anticipatory responses, and brains' heavy bias toward downward-flowing wiring (often four-to-one or even ten-to-one over inward-flowing) suggests evolution selected this as the most efficient strategy.
“we've traded the cost across time there and we're using a fairly metabolically expensive upkeep of a world model to allow a much more efficient moment by moment response”
- Stage magic works by carefully controlling attention, which the predictive brain allocates to where it expects good information; because dramatic events can occur where attention is not directed, magicians can make things appear or disappear unnoticed, demonstrating attention as a moment-by-moment balancing act in the predictive system.
“rather dramatic things can suddenly appear on stage and most people don't notice 'em because attention has been so very, very carefully controlled by the magician”
- Rather than saying perception is controlled hallucination, it is better to say hallucination is uncontrolled perception, because when the predictive process works properly it keeps you in touch with the world; hallucination occurs when the process is uncontrolled, with the brain's predictions doing all the work when disconnected from the world, while turning the dial the other way leaves predictions too weak so that faint patterns in noisy environments are missed.
“we should flip the phrase around and think of hallucination as uncontrolled perception”
Prediction error minimization explains why perception is vulnerable to illusion and why attention is essential to break established expectations.
- The predictive architecture makes us susceptible to illusions where strong predictions trump real sensory evidence, as in the hollow mask illusion where a concave mask is seen as a normal outward-facing face; people on the autism spectrum are slightly less susceptible to this and the McGurk illusion, probably because sensory inputs are enhanced relative to expectations.
“that prediction now trumps the real sensory evidence specifying concavity except it doesn't in everyone”
- Predictive brains are tuned to patterns that solved problems before and direct attention to where good information is expected, so we systematically miss outlier and unusual events, as in inattentional blindness (the gorilla experiment) and 'looked-but-didn't-see' accidents where expert drivers miss a cyclist arriving from an unexpected direction even after moving their heads that way.
“Predictive brains are tuned in to patterns that have helped them solve problems before. And so whenever we are confronted with an outlier situation, we're likely to miss it unless for some reason we're attending right there.”
- Because of predictive processing we tend not only to see what we expect to see but to read into texts what we expect to learn from them; we may believe a news article says something we strongly agree or disagree with, but careful re-reading or a different perspective reveals that meaning was brought by us rather than present in the text.
“there is a tendency to not just to see what we expect to see, which can be very damaging too but sometimes to read what we expect to read”
- The hedonic treadmill, where repeated good fortune yields no lasting happiness gain, fits the predictive account because neuronal responses to well-predicted sensory inputs are dampened, so pleasures experienced many times before are diminished unless one actively uses attention to up the sensory dial and artificially surprise oneself, as wine tasters are coached to do by letting the wine speak for itself.
“well-predicted sensory inputs cause less neuronal activity than other sensory inputs”
- Often the correct move for a prediction-driven system is to temporarily increase its own uncertainty so as to better minimize prediction errors over the long run, which gives surprise its value; we artificially curate environments to surprise ourselves, and art and science may largely be the curation of environments that harvest surprises improving our generative models.
“very often, the correct move for a prediction-driven system is to temporarily increase its own uncertainty so as to do a better job over the long time scale of minimizing prediction errors”
External resources like notebooks and smartphones function as genuine parts of cognition when robustly integrated into daily problem-solving.
- When it comes to unconscious cognition, the brain is not the limit of the machinery that counts as part of an individual's cognitive processing; external stores like a notebook or smartphone work in fundamentally the same way as biological memory because the brain cares about what information can be accessed and how fluidly, not where it is stored.
“there's no reason to think that the brain is the limit of the machinery that can count as part of an individual's cognitive processing”
- Whether to count external resources as part of the mind is not settled by functional similarity alone, which Clark sees as a stalemate; it becomes a moral or ethical decision, because treating people with dementia who rely on smartphones or their environment as merely their unaided biological capacities would do them an injustice, just as it would for someone with a well-fitted prosthetic limb.
“we're making a decision here, and that that decision, oddly enough, is in part a moral decision”
- The intermediate position that the whole body counts as part of cognition but external tools do not is unstable; if you accept that using your fingers to count makes them part of the machinery of mind, you cannot consistently deny that a constantly carried pocket calculator is part of that machinery.
“Personally I think it's an unstable space so I think you better go one way or the other”
- An external resource counts as part of the mind only if it provides robust, trusted, constantly available access woven into daily problem solving; a notebook kept in the basement that must be fetched does not count because it lacks the constant robust availability that biological memory has.
“If on the other hand you had all those numbers stored in a notebook but the notebook was in your basement and you had to run down into the basement to get it whenever you needed it, that wouldn't count because you wouldn't have the constant robust availability of that resource”
Perception and action are dual solutions to prediction error using identical neural machinery, unifying sensorimotor control.
- Motor cortex is wired pretty much like ordinary sensory cortex, so much so that many researchers avoid the sensory/motor distinction; the difference is that in motor control proprioceptive predictions play a special role, acting as motor commands, so that predicting the proprioceptive flow of reaching for an object and then minimizing the resulting errors moves the limb.
“if you look at the wiring of motor cortex, it turns out that it's actually wired pretty much like ordinary sensory cortex”
- Catching a fly ball is not done by computing the ball's trajectory and running to the predicted spot, but by running so as to keep the apparent trajectory of the ball looking motionless in the sky; doing so reliably puts you in the right place, illustrating that perception-action loops solve problems incrementally rather than all at once.
“what you do is you run in a way that keeps the apparent trajectory of the ball in the sky looking motionless as you look up there”
- Moving a limb to fit a prediction is better understood as targeted disattention than as the brain lying to itself: rather than fabricating a false belief that the arm is moving, the brain de-emphasizes the sensory information that the arm is stationary so that the goal-driven proprioceptive prediction can do its work, which also fits sports-science guidance to attend to how the body ought to be rather than how it is.
“I think it's sort of targeted disattention rather than actually lying to yourself”
- We infer that we are located wherever the perception-action loop is closing, where we perceive the world, launch interventions, and receive their effects; this explains why we feel located inside our bodies and would even feel located in the body if part of our brain were on a distant cliff top communicating wirelessly.
“we infer that we are wherever it is, that the perception action loop is closing”
Multi-level predictive expectations about body state shape perception of external events and enable placebo and reprocessing therapies.
- Even with a standard physiological cause, people's experiences of pain vary tremendously between individuals and within an individual across contexts and days in ways that do not track the organic cause, probably because different contexts activate different expectations of pain or disability that change how you feel and what you can do without any change in the underlying physiology.
“people's experiences of their pain vary tremendously, and even within a certain individual, their experiences vary tremendously from context to context and day by day in ways that just aren't tracking the organic”
- Because we are multi-level prediction machines whose high-level abstractions predict lower-level predictions down to incoming sensory signals, honest placebos work: expectations of relief can produce real relief even when patients are explicitly told they are receiving a placebo, because sub-levels of processing are automatically activated by ritual, packaging, and authoritative delivery, suggesting medicine could make more use of ritual to bring about effects through non-standard routes.
“if you're told that you're being given a placebo, it can still make a difference to the amount of relief that you feel. Presumably that's because there are all these sub-levels of processing that are getting automatically activated by good packaging and delivery by people in white coats”
- Interoceptive predictions about our own bodily state get blended with exteroceptive sensory information to structure experience of the external world, so that false cardiac feedback making people think their heart is racing causes an otherwise neutral face to be judged as angry, and in extreme cases an object reached for in a dark alley might appear as a gun when it is a smartphone.
“if you give people false cardiac feedback, so you make them think their heart's beating faster than it is, then a face that would otherwise look neutral to them is judged to be an angry face”
- Pain reprocessing therapy reframes pain not as a signal to stop an activity but as a misfiring pain-signaling system, so patients become less afraid of moving; as they do more, the brain infers the activity cannot be that harmful and the pain itself presents as lessening, replacing a vicious cycle with a virtuous one, though such reframing helps cancer-related fatigue but not the cancer or a bacterial infection.
“if you reframe it as, 'Okay, my pain signaling system is misfiring,' then you can begin to think, 'So this pain doesn't mean that I shouldn't be doing this.'”
Material symbols and physical artifacts enable thinking unreachable through internal imagination by allowing external manipulation and re-attending.
- Spontaneous gesture during speech eases certain problem solving; Goldin-Meadow and colleagues showed that children forced to explain how they solved a maths problem without being allowed to gesture perform significantly worse, suggesting gesture offloads aspects of working memory into the world.
“If children are forced to explain how they solved a maths problem, but not allowed to gesture, they're actually a lot worse at it.”
- Humans invented arbitrary systems of recombinable symbols that can be re-encountered as objects for perception, and by clothing thoughts in materiality we can attend to them in many different ways, breaking the grip of our own over-constrictive internal models, which is why building a physical scale model of an engine generates ideas you could never reach by imagining it in your head.
“by doing that, we can often break the grip of our own sort of over constrictive internal models of what kind of thing it is”
- Novels, poems, music, and even roller coasters can be understood as probability designs that lead us through building up expectations at multiple levels, cashing them out, and weighting some expectations over others; because we are multi-level prediction machines, a skilled constructor can make us surprised again and again even when re-experiencing the same work, since we cannot help being surprised even when we could have predicted what comes next.
“we should think of every novel, every poem as a probability design leading us through sort of building up expectations, cashing them out, building up at multiple levels”
- Scribbling while thinking is not merely offloading or recording thought onto a different medium; it is part of the process that actually solves the problem, captured in Feynman's reply that his notes were not a record of his thinking but the working itself.
“this isn't just offloading stuff onto a different medium. This is actually part of a process that solves the problem.”
Predictive brains are naturally curious learners that seek prediction error reduction as intrinsic reward, explaining curiosity and exploration.
- Because minimizing prediction error is itself the basic reward for predictive brains, they are naturally curious general-purpose structure learners; even with no task, they seek prediction error to minimize, which is why we will not tolerate a boring room and will invent activities, and why purely reward-driven learners find shallow solutions while prediction-driven learners explore more and, when seeded with a goal, outperform pure reward-driven agents.
“minimizing prediction error is a basic reward. So you could sort of say for brains like that, the only thing really that's rewarded, it's minimizing prediction error”
- The darkened-room worry, that a prediction-minimizing brain should hide in a dark corner where all inputs are perfectly predicted, fails because we make constant interoceptive predictions about staying alive: action is triggered long before water or glucose runs low, and even thirst-relief is itself a prediction, since drinking produces felt relief about 20 minutes before the water has physiological effect.
“there's more to the predictive brain story than just bringing exteroceptive sensory information into line with predictions. We're making all these interoceptive predictions all the time”
- Years feel faster as we age because there is less novelty in our lives; familiar experiences get sucked into well-worn troughs in our world model, producing small prediction errors, which also explains why it becomes hard to learn new things and to give new evidence its proper due late in life.
“the inputs that you are receiving get sucked into these well-worn troughs in your current world model”
Large language models lack embodied perception-action loops and thus cannot achieve genuine understanding or avoid confident hallucination.
- Large language models hallucinate with confidence partly because they are not anchored in perception-action loops; without the embodied feedback that teaches us getting things wrong has bad consequences, an LLM whose only reward is being kept alive has no reason not to confidently generate well-structured falsehoods, though prompting it to write as a well-informed expert reduces mistakes somewhat.
“their hallucinations are in a way... the result of them not being anchored in perception action loops in the right sorts of ways”
- Disembodied AI and large language models will never achieve true understanding because their grip on reality is not grounded in perception-action loops; humans are super-specialized for perception-action loops, which is what we leverage when we create material symbols and poke and prod them to think better.
“I don't think actually that those tools will ever achieve what I would call true understanding because they don't have their grip on reality grounded in perception action loops”
- If the goal is artificial general intelligence rather than a useful tool, large language models are the wrong place to start because they are trained to predict the next word in text rather than to learn basics of causation and flow in a grounded way; a developmental trajectory grounded in perception-action would be more appropriate.
“something like that has been trained to predict basically the next word in corpuses of text, and that's a very funny place to start, if what you want to be is a perception action machine”
Predictive processing accounts will eventually prove incomplete, making the important question what new understanding they point toward.
- Every story or account of how the mind works that science has had so far has turned out to be wrong, and the predictive processing account will likely prove wrong too, so the productive question is what it is a stepping stone toward.
“Every story, every account we've had so far has turned out to be wrong, and I'm sure this one will too, so the question is, what's it a stepping stone towards?”
- Academics find it hard to learn new things as they age because new inputs get assimilated to stories they already hold, so giving new evidence its proper due requires deliberate effort.
“I think us academics find that a lot as we get older.”
Cognitive scope expands across generations through symbolic culture and external technologies despite unchanging brain architecture.
- The mind of a human 100,000 years ago was very different from ours even though the brain was essentially the same, because symbolic culture and external technologies like pen and paper transformed what cognition could become.
“I would say that the mind of 100,000-year-old human was very different. But the brain obviously wasn't.”
- Friston's free energy minimization is a higher-road version of the prediction story: rather than starting from perception and action, it starts from what it takes to be a persistent living system, which must preferentially inhabit the states that define it, so a fish in a broad sense predicts it ought to be in water because its actions are all designed with that in mind.
“a high road version comes out of thinking about what it takes to be a persistent living system at all, you have to preferentially inhabit the kind of states that define you as the system that you are”