284 | Doris Tsao on How the Brain Turns Vision Into the World
What this covers
Sean Carroll hosts neuroscientist Doris Tsao in a wide-ranging conversation about how the brain converts visual input into perception and conscious experience. Tsao describes vision not as passive capture but as an active, hierarchical construction—a cascade of transformations from raw pixels in primary visual cortex through specialized regions that extract increasingly abstract features. The discussion moves from the mechanics of edge detection and face recognition into deeper questions about what consciousness is and whether explaining it requires physics beyond the known laws.
The conversation covers several distinct territories. Early portions map the anatomy of vision: the two visual streams (one for recognizing what things are, one for grasping and locating them), the discovery by Hubel and Wiesel that cortical neurons respond to edges rather than spots, and Tsao's own work decoding the neural representation of faces in monkeys with such precision that reconstructed faces cannot be distinguished from the original stimuli. The discussion then turns to how the brain solves the invariance problem—recognizing the same object across wildly different views and lighting conditions—and explores surprising findings, such as evidence that humans repurposed face-recognition cortex to recognize letters. A significant portion addresses consciousness itself: whether it arises continuously or in discrete frames, how predictive coding might explain why perception feels unified, and whether the philosophical zombie argument poses a real puzzle or dissolves under physicalist analysis. Both speakers argue that consciousness is an emergent property that requires no new physics to explain.
Tsao argues that vision is not passive pixel-capture but an active, hierarchical construction in which specialized cortical regions transform raw input into objects, faces, and a generated model of the world—and that consciousness itself is likely a discrete, top-down generative process that will ultimately be explained mechanistically rather than requiring new physics.
- The visual cortex is a hierarchical factory (V1 edges -> ventral stream object recognition) with distinct dorsal/ventral streams dissociated by lesion evidence
- Conscious perception is generated top-down (predictive coding) to be consistent across representational levels, and is sampled in discrete frames rather than continuous
- Both Tsao and Carroll hold that consciousness is an emergent property of complex systems and the 'hard problem' will dissolve rather than require new laws of physics
Visual cortex contains dozens of specialized areas organized into distinct hierarchical pathways that transform raw sensory input into coherent object perception.
- The eye works like a camera—light passes through the lens onto photoreceptors and signals travel via the optic nerve to the visual cortex—but the visual cortex is a massive piece of machinery (about a third of the monkey brain) containing dozens of specialized areas that transform raw pixels into a perception of objects in space.
“the visual cortex is this incredible piece of machinery. In monkeys, it probably takes up about a third of the brain... there's dozens of different areas that are specialized for processing different aspects of the visual world. So it's like a whole factory that's transforming these pixels into your perception of objects in space.”
- There are two distinct physical pathways of visual processing: the ventral stream specializes in object recognition (identifying what something is, like a leopard or your mother), while the dorsal stream is the pathway to motor cortex, involved in knowing where things are and their 3D shape so they can be grasped correctly; lesion evidence shows people can selectively lose the ability to recognize objects while still grasping them, or vice versa, proving these functions are physically dissociated.
“there's actually two streams of visual processing. There's this dorsal stream and this ventral stream... the ventral stream is specialized for object recognition... the dorsal stream is really the pathway to our motor cortex... it's involved in knowing where things are and what their 3D shape is so that we can grasp them”
- Inferior temporal (IT) cortex is the end of the ventral stream and is responsible for object recognition—identifying objects (your mom, keys, a pencil) invariant to how they are presented (profile, frontal, or back of the head)—which is a huge computational problem that IT cortex appears to solve.
“inferior temporal cortex... that's the pathway that's responsible for object recognition... recognizing them invariant to how they're presented... That's a huge computational problem. And I think inferior temporal cortex is the part of the brain that's solving that problem of object recognition.”
- Visual processing is not a one-way flow; there are abundant feedback connections, and almost every single connection in the brain is bidirectional—with the IT cortex to striatum link being one of the rare exceptions that goes only one way.
“There are so many feedback connections... almost every single connection in the brain is bidirectional.”
- Hubel and Wiesel followed the anatomy of retinal ganglion cells into the brain: in the lateral geniculate nucleus cells still showed center-surround responses like the retina, but in primary visual cortex (V1) the cells no longer responded to spots of light at all and instead required edges at specific locations in the visual field—revealing a dramatic transformation in how visual information is represented.
“they went one stage further into primary visual cortex, and there they uncovered this whole new world... The cells suddenly didn't respond to spots of light at all, but they required edges... different cells respond to edges at different locations in the visual field.”
Face recognition depends on dedicated, multidimensional neural machinery that solves view-invariant object identification and evolved from shallow hierarchies that deepened over time.
- Lesion studies showed patients who lose only the ability to recognize faces while recognizing everything else fine, and in 1997 Nancy Kanwisher's landmark paper found a blueberry-sized region in the right temporal lobe (the fusiform face area, in the fusiform gyrus) that responded far more to faces than other objects and was in the same place in every subject—strong evidence for a piece of cortex dedicated to representing faces.
“in 1997, Nancy Kanwisher at MIT published this landmark paper... she reported that human patients... showed them pictures of faces and objects. They all showed this region in their temporal lobe... that responded much more to faces than other objects. And it was like the size of a blueberry. And it was like in the same place in every single subject.”
- Faces are multidimensional—requiring at least 50 to 200 dimensions to create a really good likeness—and the brain has specialized machinery to see all those dimensions simultaneously (compared to color's three dimensions); the fusiform face area processes the physical features (inter-eye distance, texture, shape) rather than emotion or explicit identity, which are handled later.
“faces are multidimensional, right? You need at least like 50 or 200 dimensions to create a really good likeness. And we can see all those dimensions simultaneously... color has three dimensions.”
- By recording the neural activity of about 200 neurons in the face areas, Tsao's lab was able to reconstruct precisely the face a monkey was seeing, creating a likeness so accurate you couldn't tell which was the real stimulus and which was the reconstruction—demonstrating that the neural code for faces can be decoded.
“take just the neural activity from 200 neurons, and from that be able to reconstruct precisely the face that the monkey was seeing and creating a likeness that you couldn't even tell which one was the real stimulus and which one was the reconstruction that we made from his neural activity.”
- Monkeys have six face areas (patches), each performing a different function arranged hierarchically: the most posterior patch likes eyes (a dark disc within an outline), the next cares about faces at specific views, more anterior patches respond mirror-symmetrically (left and right profiles alike), and the most anterior patch contains cells that respond in a view-invariant way—firing for the same identity regardless of facial orientation.
“not only did they have a face area, they had six of them... the most posterior one, it seems to like really like eyes... the next one... cares about faces, but at specific views... it responds to faces in a mirror symmetric way... the most anterior patch, you have these incredible cells that respond in a view invariant way”
- In tree shrews—close non-primate relatives of primates with excellent high-acuity vision—Tsao's lab found the surprising result that V2 (the area right after V1) seems to perform many of the functions of IT cortex, including containing face cells, suggesting visual object recognition evolved from a shallow hierarchy that became deeper over evolutionary time.
“we found this very surprising result that V2... the area right after V1 seems to have many of the functions of IT cortex. And there's even face cells in tree shrew V2... we think that it started from a very shallow hierarchy and became deeper.”
Consciousness arises in the brain through generative, predictive models that construct discrete conscious frames spliced with unconscious gaps, not through continuous perception.
- Under the predictive coding model, perception is a cascade of dominoes: thinking you see a face in the clouds biases V2 to generate the corresponding edges, which feeds back to V1 telling it a region should be a bit darker because that's the eye, with V1 actually filling that in—though after five years of work Tsao's lab still considers the evidence for this generative feedback hypothesis unproven.
“you think you see a face in the clouds, then that's going to bias V2 to generate these edges. And then it's going to go back to V1 and say, 'Oh, that should be like a little bit darker because that's the eye.' And V1 would actually be filling that in.”
- Although consciousness feels continuous, perception is actually constructed in discrete time points where the brain integrates accumulated measurements into what we consciously see; moreover these conscious frames are not consecutive like a film strip but are spliced with epochs—up to a few hundred milliseconds—during which the neurons are not representing what you consciously see, so you are effectively unconscious in between and never notice the gap.
“we think that inference step where you put all this together and then build what you consciously see, we think that occurs in discrete time points. It's actually, even though consciousness feels like it's continuous, we think it's happening discreetly.”
- Under the predictive coding (or generative model) theory of consciousness, all sensory input is processed to tell the brain what to generate, but what we are consciously aware of is the top-down process that recreates the world; this elegantly explains why conscious perception is always consistent across different representational levels—e.g. in the face/vase illusion, every edge detail (coded in V2) is consistently owned by whichever object you perceive, even though the neurons coding the left and right edges never directly communicate, because the percept is generated to be consistent.
“everything that we're conscious of is generated by the brain. So all this stuff goes in and we process it, and it gives us the input to know what to generate. But then what we actually are consciously aware of is the top down process that actually recreates the world.”
- As AI matures and becomes conscious, we will be able to create new qualia and discover the laws that govern the relationship between physical configuration and subjective experience, at which point the mystery will completely dissolve and consciousness will be seen as a basic property of complex systems.
“as AI matures and becomes conscious we'll be able to create these new quality and we'll really discover the laws that govern this relationship. And like you say, it's gonna completely dissolve away. And we'll see that consciousness is, this basic property of complex systems?”
The brain solves the invariance problem—recognizing persistent objects despite changing input—using compression and dynamic receptive fields rather than pixel-by-pixel rewriting.
- The biggest job of the visual system is solving the invariance problem—figuring out what corresponds to the same thing as input changes drastically—which functions like a compression problem: when the gaze shifts slightly, rather than rewriting the entire huge visual field, the brain represents the scene as almost the same thing shifted by a small amount, an enormous saving in energy and information processing achieved via dynamic receptive fields that transform to counteract image changes.
“The biggest job of the visual system is to solve this in variance problem and figure out what is corresponding to the same thing, 'cause the information is changing so much”
- Object identity across viewpoints can be solved without training data using differential geometry: a surface is defined as a collection of overlapping charts, so as you move around an object you compute one chart, change perspective (via two eyes or motion), find an overlapping chart, and form an equivalence class of charts whose equivalence relation is overlap—giving an elegant mathematical theory of how persistent objects arise, in contrast to computer vision's reliance on massive training data.
“people in computer vision, they say you just put lots of samples... And what we say in the paper is that no, you don't need any training data. It's actually... a really elegant mathematical problem that goes to the very definition of a surface... a surface is this collection of charts, overlapping charts... I can form this equivalence class of charts, where the equivalence relation is to overlap.”
- The origin of abstract or symbolic thought rests on the concrete problem of segmenting and tracking persistent objects: before the brain can assign labels, associate, and reason about things, it must transform the stream of pixels into discrete persistent objects; until we have the concept of objects, we cannot think about anything—it is just sensory blur—making object segmentation the key, still poorly understood, step toward symbolic cognition.
“I see the big challenge to understanding how symbolic thought arose as understanding this very concrete problem of segmenting and tracking... Before we have that concept of objects, we can't even think about anything, right? It's just like this sensory blur. So I think that's like the key step. And we don't understand that at all.”
Behavioral sophistication and representational complexity sufficiently determine consciousness; philosophical zombies and conscious beings with identical behavior would be equally conscious.
- Since genes care only about behavior and not about whether we are conscious, a philosophical zombie would propagate its genes exactly as a conscious being would; therefore either we are incredibly lucky that brains with our behavior also happen to be conscious, or—more plausibly—any sufficiently complex system that represents and navigates the world as sophisticatedly as we do, with all our behaviors, is likely going to be conscious.
“we evolve to survive, right? Our genes don't care at all if we're conscious or not... if we're like a pea zombie, our genes would be propagated exactly the same as if we're conscious. Therefore... either consciousness is like, we're just incredibly lucky... or any kind of system... that does what we are capable of doing... Anything that has all of our behaviors is likely going to be conscious.”
- The brain operates like a video game engine: there is a space of possible reality that percepts must live in, and all the incoming sensory inputs do is turn the knobs that determine which reality the brain generates.
“the brain has like something like a video game engine. Like there's just a space of reality that it has to live in... And all the inputs are doing is turning the knobs and then you generate that. Reality.”
Face-processing areas exhibit neural plasticity, reorganizing for different functions across species and being repurposed for letter recognition in literate humans.
- The Thatcher illusion—where facial features are turned upside down within an upright face frame, looking freaky, but appearing normal once the whole face is inverted—works because inverting the face essentially silences the face-processing areas, which are not wired to respond to upside-down faces; this lets a person experience the equivalent of brain damage while still feeling they see everything, with nothing apparently missing.
“what's happening when you turn it, that face upside down is that you're essentially inducing a lesion in your... You're causing your face areas to become silent, 'cause they're just not wired to respond to upside down faces. And so you can experience what it's like to have brain damage. And what's remarkable is that you feel like you see everything.”
- The face area sits in the right hemisphere, and the corresponding piece of cortex in the left hemisphere is responsible for recognizing letters; since monkeys and illiterate humans have bilateral (symmetric) face areas, the conclusion is that humans repurposed cortex that originally recognized faces to recognize letters, demonstrating remarkable plasticity in the human visual system.
“this face area in the human is in the right hemisphere, and the corresponding piece of cortex in the left hemisphere is actually responsible for recognizing letters... in illiterate people who don't recognize letters, they also have bilateral face areas.”
- Some people have prosopagnosia (trouble recognizing faces) from lesions, but a high percentage—around 4% of the population—are very bad at recognizing faces without ever having had a stroke, and brain studies show some have poor selectivity or small face areas.
“there's like 4% of the population, some very high percentage didn't have any stroke, but they're just like really, really bad at recognizing faces. And indeed, if you study the the brain activity in these people, some of them have poor selectivity in their face areas or small face areas”
Visual motion and illusions reveal how direction-selective cells and upside-down face selectivity shape conscious perception in elegant and counterintuitive ways.
- A sharp, fine edge can only be represented at high enough resolution in primary visual cortex (V1), yet we are consciously aware of it—raising the puzzle of whether V1 itself is conscious, even though everything we understand about V1's processing (edge detection) seems mundane and programmable, which makes it mysterious what exactly confers consciousness on it.
“the only part of the brain where the neurons are, have the high enough resolution to represent the edge is probably V1... And yet you're conscious of it. Like, how is that possible? Does that mean that like V1 is conscious?... like, what makes that conscious?”
- Neuroscientists firmly believe that deep neural networks were inspired by Hubel and Wiesel's discoveries about the layered, hierarchical processing in visual cortex.
“I think the neuroscientists firmly believe that deep neural networks were inspired by Hubel and Wiesel's discoveries about visual cortex.”
- The reverse-phi illusion, in which a bicycle can be made to look continuously moving simply by setting the contrast correctly, is beautifully explained by the properties of direction-selective cells in the motion-processing part of the brain.
“there's this very cool illusion called reverse phi where you can make a bicycle look like it's constantly moving just by setting the contrast in the correct way. And that is beautifully explained that by the properties of the direction selective cells in the motion processing part of the brain.”
- One of the most beautiful things about being a vision scientist is that you open your eyes and see the very miracle you are trying to explain, so in some sense everyone is an expert on vision.
“you open your eyes and you see the miracle that you're trying to explain, like we're all experts on vision in some sense.”
- If you ask what fraction of the human brain responds to a visual stimulus, it is essentially the whole brain—when subjects watch movies in an fMRI scanner the whole brain lights up, much of it multimodal but still responding to visual input.
“if you ask what fraction of the human brain will respond to a visual stimulus, it's basically the whole brain. Jack, my colleague Jack Allen puts people inside FRS scanners and shows them movies and the whole brain lights up.”