Adam Marblestone – AI is missing something fundamental about the brain
What this covers
Adam Marblestone and Dwarkesh Patel examine why current AI systems may lack something fundamental that the brain possesses. Marblestone proposes that the critical difference lies not in architecture but in how the brain encodes learning objectives. He argues the brain splits into two systems: a Steering Subsystem implementing many evolutionarily-tuned loss functions, and a Learning Subsystem (the cortex) that learns general skills. This arrangement allows the genome to specify high-level desires compactly—as evolutionary code rather than as pre-trained network weights. Large language models, by contrast, use mathematically simple loss functions and omit the value functions that appear central to biological Model-Based vs Model-Free RL. The conversation traces how Steve Byrnes's proposal explains the mechanics: the Steering Subsystem uses innate sensors (like the superior colliculus) to detect threats or reward-relevant features, training cortical Thought Assessors that wire learned abstract concepts to these innate responses.
The discussion moves across neurobiology and AI practice. Marblestone draws on single-cell atlas work showing that subcortical regions contain far more diverse, bespoke cell types than the cortex—evidence that the Steering Subsystem implements genetically-wired specificity while the cortex needs only a repeating architecture. He addresses connectomics infrastructure, defending connectome work not as a path to reverse-engineer individual circuits but as constraint-generation for computational hypotheses. The conversation circles back to what should be described: rather than interpreting learned features, neuroscience and AI should adopt what Marblestone calls Describe-by-Recipe vs Interpret-the-Weights—characterizing systems by their architecture, learning rule, and training regime. Throughout, the speakers note that researchers dismissed as outdated—those favoring Energy-Based Models and stochastic, sampling-based inference—may have grasped something the recent LLM consensus overlooked, making the brain's approach to reward specification potentially decisive for AI alignment and capability.
Marblestone argues that the brain's 'secret sauce' is not its architecture but a vast set of evolutionarily-tuned, species-specific reward and cost functions implemented in a subcortical Steering Subsystem, which bootstraps a general Learning Subsystem (cortex) — explaining how the genome encodes high-level desires with little information and why current LLMs may be missing something fundamental.
- Evolution encoded complexity into many specific loss/cost functions rather than into a pretrained network, which is why the genome can specify intelligence compactly.
- A subcortical Steering Subsystem with bespoke cell types and its own sensory system trains predictors (Thought Assessors) in the cortex, wiring learned world-model features to innate rewards.
- LLMs use mathematically simple loss functions and a degenerate form of RL without value functions, unlike the brain.
This asset isn't compiled yet
You're seeing its claims, ranked. Compile it to build the argument threads, weight them, and check each claim against your library — the full view.
The brain layers multiple RL schemes: the striatum/basal ganglia perform simple model-free RL over a finite action space; dopamine carries a reward prediction error signal (per Peter Dayan) consistent with learning value functions; and the cortex builds a model-based world model that predicts when rewards occur, and via RL-as-inference can clamp reward high and sample plans likely to achieve it.
“there are some parts of the brain that are thought to do something that's very much like model-free RL, that's parts of the striatum and basal ganglia.”
The reason to believe AI-derived vocabulary applies to the brain is empirical: AI models built on these ideas work surprisingly well, and the best computational neuroscience models (e.g. CNNs predicting visual cortex activations on DiCarlo's brain score) outperform native neuroscience models — TD learning, independently invented by Sutton, also appears in dopamine signaling — though critics like Buzsáki argue we should instead derive the brain's true primitives bottom-up.
“the best computational neuroscience theories we have seem to have been invented largely as a result of AI models and finding things that work.”
The AI field has neglected the role of very specific loss and cost functions; whereas machine learning prefers mathematically simple loss functions (next-token prediction, cross-entropy), evolution built enormous complexity into many different loss functions, turned on at different developmental stages, effectively a curriculum encoded as 'Python code' for what each brain region must learn.
“My personal hunch within that framework is that the field has neglected the role of these very specific loss functions, very specific cost functions. Machine learning tends to like mathematically simple loss functions.”
Channeling Byrnes, the minimum set of Steering Subsystem components needed to produce something intelligent is far smaller than the set needed for human-like social instincts and ethics; therefore it is probably possible to build powerful model-based RL optimizers (potential paperclip maximizers) lacking most human reward functions, which is why understanding the Steering Subsystem matters chiefly for alignment.
“he's very concerned that the minimum viable things in the Steering Subsystem that you need to get something smart is way less than the minimum viable set of things you need for it to have human-like social instincts and ethics and stuff like that.”
Steve Byrnes proposes a solution to Ilya Sutskever's question of how evolution encodes high-level desires: a Steering Subsystem with innate reward functions and primitive sensors (e.g. superior colliculus detecting faces/threats) trains predictors ('Thought Assessors') in the cortex/amygdala that learn which abstract world-model neurons predict innate responses, thereby robustly wiring learned features to innate rewards.
“The basic idea that Steve Byrnes is proposing is that part of the cortex, or other areas like the amygdala that learn, what they're doing is they're modeling the Steering Subsystem.”
Amortized inference replaces the intractable full Bayesian process — sampling over possible causes via Monte Carlo to compute a posterior — with a trained feedforward map from observation directly to the most likely cause, trading exact inference for speed; test-time/inference-time compute (chain of thought) effectively reverts to the slow sampling process, and capabilities discovered that way later get distilled back into the base model.
“You don't have to evaluate all these energy values or whatever and sample around to make them higher and lower. You just say, approximately that process would result in this being the top one or something like that.”
E11 Bio's optical-microscopy approach yields a 'molecularly annotated connectome' — not just who connects to whom, but which molecules and receptor combinations are present at each synapse and what cell type each neuron is — because photons (unlike electrons) don't damage tissue, allowing gentle washing and imaging of fragile molecules, with activity mapping then used to train ML models predicting function from structure.
“with E11's approach, you can get a "molecularly annotated connectome." That's not just who is connected to who by some synapse, but what are the molecules that are present at the synapse?”
Single-cell atlas work (Fei Chen, Evan Macosko, BRAIN Initiative) shows that subcortical Steering Subsystem regions like the hypothalamus contain far more diverse and bespoke cell types than the cortex, consistent with the idea that the cortex needs only enough cell types for a repeating learning architecture while the Steering Subsystem needs many species-specific cells to implement innate reward circuits via genetic wiring.
“there are a lot more weird and diverse and bespoke cell types in the Steering Subsystem, basically, than there are in the Learning Subsystem.”
If a major part of intelligence is the reward/loss functions that aid sample-efficient learning, then the small information content of the genome (~3GB, only a fraction coding the brain) is explained: a reward function is just a line of Python, so a thousand such lines plus a compact learning algorithm and reusable architecture can specify an intelligence without storing a pretrained network.
“Because the reward function, in Python the reward function is literally a line. So you just have a thousand lines like this, and that doesn't take up that much space.”
Rather than chasing interpretability of individual learned features (like the 'Golden Gate Bridge' circuit), which will be hopelessly complex, we should describe the brain the way we describe an LLM — in terms of architecture, learning rule, initialization, hyperparameters, and training data — because (per Kording & Lillicrap) a network trained to compute arbitrarily complex things can never be fully interpreted, but the recipe that produced it can be understood.
“What I think we should do is we should describe the brain more in that language of things like architectures, learning rules, initializations, rather than trying to find the Golden Gate Bridge circuit.”
Following the Human Genome Project pattern — where structured competition and a paradigm shift from macroscopic chemistry to massively parallel single-molecule imaging dropped sequencing cost a million-fold in a decade — E11 Bio is shifting connectomics from electron microscopy to optical microscopy to cut a single mouse-brain connectome from billions of dollars toward low tens of millions, with a human brain (~1000x larger) potentially achievable for under a billion dollars.
“E11 technology, and the suite of efforts in the field, is trying to get a single mouse connectome down to low tens of millions of dollars.”
Because Lean lets a proof be mechanically verified as following perfectly from its assumptions, formalized math becomes a perfect RLVR (reinforcement learning from verifiable rewards) signal, so AI will keep improving at searching for and finding proofs — analogous to AlphaGo searching the game of Go — automating the mechanical parts of math (validating lemmas, checking equivalences) while leaving conjecturing and conceptual organization to humans.
“it also makes it easy for correctness of proofs to be an RL signal in very much RLVR... That becomes a perfect RLVR task.”
The same Lean proof machinery that proves theorems can be used for formally verified software — mathematically proving that code cannot be hacked in certain ways, that memory regions cannot affect each other, or that it has certain properties — which Marblestone argues will become a powerful cybersecurity tool, and that an AI able to prove the Riemann hypothesis could prove insanely complex software properties and synthesize provably-correct software on demand.
“you can write math proofs about software and say, "This code, not only does it pass these unit tests, but I can mathematically prove that there's no way to hack it in these ways."”
The cortical predictor of innate responses can generalize because its input is the rich, abstract world model: even the word or concept 'spider' (a book about spiders, a room with spiders) can trigger the spider-flinch reward circuit, whereas the Steering Subsystem itself can only respond to the limited primitive features its own sensors (superior colliculus) can detect.
“The cortex inherently has the ability to generalize because it's just predicting based on these very abstract variables and all these integrated information that it has. Whereas the Steering Subsystem only can use whatever the superior colliculus and a few other sensors can spit out.”
The fast expansion of the hominid brain is explained because enlarging the cortex requires very few additional genes — it largely replicates the existing repeating cortical architecture (like making an eight-layer transformer from a six-layer one) — and increased social learning raised the returns to a bigger cortex by improving sample efficiency (learning from an elder instead of trial-and-error).
“that can be done with a relatively few genes, because it's really replicating what the mouse already has, making more of it.”
There may be a hierarchy of nested model-free optimization — evolution (extremely unforesighted) producing brains, basal ganglia doing model-free RL, cortex doing model-based learning, and culture doing model-free-like group selection — suggesting that very simple, unforesighted algorithms can produce arbitrarily sophisticated outcomes if run at enough scale.
“Maybe the simple algorithms can just get you anything if you do it enough.”
Innate reward functions cannot rely on plasticity-driven learning; they need either special cell types or special genetic wiring rules whereby cells express matching receptors and proteins so that 'when this cell contacts that cell, form a synapse', which is why bespoke cell types are needed to specify Steering Subsystem circuitry without learning.
“those cells express different receptors and proteins that say, "Okay, when this one comes in contact with this one, let's form a synapse." So it's genetic wiring, and those need cell types to do it.”
Marblestone rejects the radical view that memory/learning is mostly genetic or non-synaptic; the abundant intracellular machinery (e.g. for weight normalization or memory consolidation) is mostly implementing, in molecular form without a central controller, what a digital computer does trivially by editing weights — though some cases, like cerebellar cells storing a time-delay constant in the cell body, do appear to compute something genuinely beyond synaptic connectivity.
“algorithmically, it's not really adding something beyond these algorithms. It's just implementing something that in a digital computer is very easy for us to go and just find the weights and change them.”
Solving the creative part of math requires a loss function for good conjectures: an interesting theorem is one that compresses lots of information and implies many other theorems via short inference chains, so a powerful proof can be framed in terms of how much it reduces the Kolmogorov complexity of generating the rest of the network of proofs — and some researchers (e.g. Bengio) are attempting to formalize such a loss for good explanations.
“A really interesting math proof or statement might be one that compresses lots of information and has lots of implications for lots of other theorems.”
A key disadvantage of biological neural hardware is that it cannot be copied or given external read-write access to every neuron and synapse, unlike digital weights which can be edited and copied in code; but biological hardware may have offsetting advantages such as co-located memory and compute, low-voltage energy efficiency, and naturally stochastic neurons that supply free sampling for probabilistic inference.
“an obvious downside of the brain is it cannot be copied. You don't have external read-write access to every neuron and synapse, whereas you do.”
A connectome is valuable not because it directly reveals the learning algorithm, but because it supplies an enormous number of additional constraints — how many dopamine signal types exist, whether prefrontal-to-auditory wiring matches prefrontal-to-visual wiring — that refine the space of candidate AI-style hypotheses (energy-based model vs amortized VAE, local vs global learning rules) far more efficiently than bespoke one-off experiments.
“just think of the connectome as a huge number of additional constraints that will help to refine, to ultimately have a consistent picture of that.”
Continual learning in the brain may arise from architectural features (hippocampal sequence storage and replay consolidating memories into cortex) combined with multiple timescales of plasticity — neurons doing fast-weight plasticity and slower backprop-like plasticity simultaneously, or synapses with many states — rather than from a single fundamental change to the learning algorithm.
“There might be multiple timescales of plasticity or clever learning rules that can simultaneously be storing short-term information and also doing backprop with it.”
In a world where AGI is far away but 'automated cleverness' plus formal verification exists, it may make sense to turn nearly everything AIs do into provable symbolic statements, enabling billions of AI agents to safely build on each other — proofs guard against exploitation and brain-hacking messages — pointing toward davidad's safeguarded-AI vision of explicitly interpretable, equation-based world models rather than opaque neural-net world models.
“the only way the future AGI civilization can collaborate with each other is if they can prove each step.”
The neuroscience perspective suggests that researchers often mocked online — Yann LeCun and Beff Jezos with their probabilistic/energy-based model views — may be onto something, since stochastic neurons, sampling, and probabilistic inference are what nearly all neuroscientists and AI people believed until around 2021, before LLM scaling shifted the consensus.
“I think Yann LeCun and Beff Jezos are kind of onto something about the probabilistic models or at least possibly. In fact that's what all the neuroscientists and all the AI people thought until 2021 or something.”
Because digital minds can be copied while biological minds cannot, the tradeoffs differ: it makes economic sense to amortize more capabilities into copyable digital weights, whereas evolution had to re-pass everything through the genome and rebuild each network within a dynamic lifetime, so it amortized less into the Learning Subsystem and more into innate behaviors and bootstrapping cost functions.
“digital minds which can be copied, have different tradeoffs which are relevant, from biological minds which cannot. So in general, it should make sense to amortize more things because you can literally copy the amortization.”
Unlike an LLM which natively computes one conditional probability (next token given prior context), the cortex may be a general prediction engine in which any area can predict any subset of its input variables from any other subset — omnidirectional inference — more akin to energy-based models that represent the joint distribution and allow clamping arbitrary variables and sampling the rest.
“What if the cortex is natively made so that any area of cortex can predict any pattern in any subset of its inputs given any other missing subset? That is a little bit more like "probabilistic AI".”
Marblestone's hunch is that the brain's world-model representation is not a clean symbolic language — true symbolic operations like variable binding (copying a variable between registers regardless of content) may not have dedicated machinery; instead cost functions and architectures may make such behavior approximately emerge much as it might in a neural net — so the representation is likely 'a huge mess'.
“my hunch is that it's going to be a huge mess and we should look at the architecture, the loss functions, and the learning rules. I don't expect it to be pretty in there.”
An idea (originally from a Gwern blog post) is to train AI not only to predict labels but also, as an auxiliary loss, to predict the human neural activity pattern evoked by the stimulus, potentially sculpting the network to represent information the way the brain does, improving generalization and robustness (e.g. to adversarial examples) — limited today only by the difficulty of collecting brain-activity data, not by principle.
“If you add that as an auxiliary loss function or an auxiliary prediction task, does that sculpt the network to know the information that humans know about cats and dogs and to represent it in a way that's consistent with how the brain represents it.”
Marblestone's meta-level take is that answering the deep questions about the brain will not come merely from smart people thinking harder, but requires empowering neuroscience to become a far more technologically powerful, ground-truth science — chiefly through comprehensive, unbiased data collection like whole-brain mapping rather than bespoke one-off experiments.
“we have to empower the field of neuroscience to just make neuroscience a more powerful field technologically and otherwise, to actually be able to crack a question like this.”
From mapping fundamental capability 'gaps' across science, Marblestone found that essentially every field — even mathematics, which he assumed needed only whiteboards — is missing scalable infrastructure (like Lean for math, connectomics for neuroscience), and that the total set of a few hundred such fundamental capabilities would cost only low billions of dollars to address, far less than expected.
“some amount of scalable infrastructure is missing in essentially every area of science, even math, which is crazy.”
Current LLM RL is conceptually a very dumb form of RL — every token in a successful trajectory is upweighted — and notably lacks value functions, even though far more sophisticated methods like temporal difference / Q-learning existed a decade ago (e.g. Atari game-playing); it is surprising that this crude approach works as well as it does.
“The fact that we don't have value functions at all in the LLMs is crazy.”
Just as Steve Byrnes, an outsider AI-safety researcher, could synthesize the neuroscience literature into the Steering/Learning framework, formal proof tools could let outsiders contribute to fields like string theory — an 'outsider string theorist' whose math is handled by the computer — potentially increasing innovation by lowering the barrier of mechanical expertise.
“Can you have outsider string theorists or something, because the math is just done for them by the computer? And does that lead to more innovation in string theory? Maybe yes.”
The practical relevance of doing brain-mapping science now depends heavily on AI timelines: in a fast 'AI 2027' scenario the science is obviated because the transformative event determines the science of ten years out, but Marblestone leans toward ~10-year timelines, partly because LLMs look weirdly different from brain mechanisms and he is waiting for AI that resembles brain-like model-based RL.
“it's not super practical and relevant if you're in an AI 2027 scenario. And so what science I'm doing now is not going to affect the science of ten years from now.”
How the brain learns so much from so little may be the most important question in science.
“You can make an argument that this is the most important question in science.”