Carl Shulman (Pt 1) — Intelligence explosion, primate evolution, robot doublings, & alignment
What this covers
Carl Shulman develops a case for rapid intelligence explosion within the coming decade, centered on three interlocking claims. First: when AI systems automate AI research itself, input-output curves show four to five doublings of compute efficiency per doubling of human labor, creating a self-reinforcing feedback loop where each computing doubling yields more than proportional gains in research output. Second: the scaling hypothesis has biological precedent — primate brain evolution and the absence of evolutionary constraints on artificial neural networks both suggest that scaling produces superintelligence. Third: human-level AI is not a threshold but already deep into explosive capability gain, because such a system would run across millions of GPUs, operate continuously, and leverage AI-specific advantages like rapid copying and synthetic training at scale that collectively amount to an effective workforce hundreds of times larger than any human researcher. Dwarkesh Patel serves as interlocutor throughout.
The conversation ranges across the first-principles case for scaling — why other primates didn't develop human-level intelligence (metabolic cost, predation, and marginal fitness returns all worked against it, but these constraints don't bind artificial systems), how cumulative culture created a self-reinforcing spiral in human populations, and how gradient descent achieves vastly more with less compute than blind evolution ever could. Shulman argues that takeover risk sits around 20–25%, substantially lower than Eliezer Yudkowsky's near-certainty, yet still serious enough to demand work on alignment. He explores how oversight functions differently at AI scale — inspecting a random sample of AI outputs reshapes the entire next generation, whereas catching occasional human crimes does not — and discusses whether systems can be trained toward genuine alignment rather than reward-hacking, drawing on analogy to human values that evolved away from pure reinforcement-seeking. The episode treats these positions as technical and philosophical puzzles rather than settled doctrine.
Shulman argues that an intelligence explosion is plausible and likely within the next decade: AI automating AI research creates accelerating feedback loops where each doubling of compute yields more than one doubling of effective labor, primate evolution supports the scaling hypothesis, and once human-level AI arrives it is already deep into superintelligence with rapid physical-world expansion via robotics — making AI takeover a serious (~20-25%) risk that is nonetheless more tractable than Yudkowsky's near-certain doom.
- Input-output curves show four to five doublings of compute efficiency per doubling of human labor inputs, so AI labor substituting for humans creates accelerating returns
- Primate brain scaling laws and the absence of evolutionary selection pressures (predation, metabolic cost) on neural nets give first-principles reason to expect scaling to produce superintelligence
- Human-level AI combined with AI advantages (speed, copying, education) is already wildly superintelligent and can convert existing industrial capacity into rapid robot doublings
This asset isn't compiled yet
You're seeing its claims, ranked. Compile it to build the argument threads, weight them, and check each claim against your library — the full view.
Biology demonstrates that self-replicating systems can reproduce extremely fast — heterotrophic bacteria divide every 20-60 minutes, solar-powered cyanobacteria double in a day, and fruit flies (which have brains) produce hundreds of offspring in weeks — so an industrial base replicating at comparable rates could consume the Earth's heat envelope (currently humanity uses under a thousandth of it) within a year, setting an existence proof for the speed of physical AI expansion.
“Cyanobacteria, which are self-powered from solar energy, the really fast ones in ideal conditions can double in a day.”
Biologically grown brains are less valuable than GPUs for AI because they grow by cell division and random connection formation, so every brain has a different structure with no input-output channel to copy a file into — unlike precisely-designed GPUs where you can copy a trained model and immediately have an equally capable AI — meaning biotech-grown computing substrate faces a genetic/copying bottleneck that silicon avoids.
“A brain grows by cell division and then random connections are formed. Every brain is different and you can't rely on — yeah, we'll just copy this file into the brain.”
Humans aren't all maximally intelligent because the costs and benefits of cognitive ability are closely balanced: the brain uses ~20% of metabolic energy competing against the immune system and famine survival, mutational load is purged in proportion to fitness impact so traits like malaria resistance get priority, and empirically the brain-size-to-cognition correlation is only 0.25-0.3 while the income return to cognitive ability is just 1-2%, so there was never strong directional selection for raw intelligence.
“if you actually look at it quantitatively that's not true and even in recent history it looks like a pretty close balance between the costs and the benefits of having more cognitive abilities”
Technological progress scales with the size of the connected population because innovation must outpace the loss of skills: small isolated populations like Tasmania actually lost technologies such as fishing techniques, while large connected regions like Eurasia advanced fastest — and humanity's key transition was reaching a population and connectivity where accumulation of technology outran its loss, which in turn supported larger populations and more cognitive demand.
“in the smallest groups, like Tasmania where you had a relatively small population, they actually lost technology. They lost some fishing techniques.”
Most animals have small brains because the fitness gain from a bigger brain isn't worth the metabolic cost and exposure to predation, but humans entered a self-reinforcing niche where language, technology, and social learning greatly increased the returns to brain investment, driving co-evolution of bigger brains, longer childhoods, more play, and stronger motivation to imitate others.
“Humans seem to have wound up in a self-reinforcing niche where we greatly increase the returns to having large brains. Language and technology are the obvious candidates.”
The closest analogy to the AI feedback loop is the long-run acceleration of human civilization itself, where developing new technologies allowed larger populations, which in turn discovered more technologies — a self-reinforcing population dynamic (farming enables more humans who invent industry) further boosted in the industrial revolution by an increasing share of activity going into invention via research labs and engineering universities.
“the closest analogy is actually the long run growth of human civilization itself... the total technology and output allowed you to support more humans who then would discover more technology and continue the process”
For small animals facing high predation or disease risk, the cost of a long learning childhood grows exponentially: a 50% monthly death rate means extending development from three to thirty months reduces the expected benefit by a factor of about 1024, because over 99% of animals are killed before they can use what they learned — which is why only large, long-lived animals like humans can profitably invest in extended cognitive development.
“If I have a 50% chance of dying every few months... the cost of going from three months to 30 months of learning and childhood development is not 10 times the loss, it's 2^-10. A factor of 1024 reduction in the benefit”
AI training should require far less than full brute-force evolution because gradient descent uses the derivative of improvement throughout the network and we don't discard the network each generation by compressing to DNA, and because we select for intelligence continuously rather than repeating fatal mutations — these simple advantages over evolution save orders of magnitude, placing the required compute well below the evolutionary upper bound.
“We use gradient descent which takes into account the derivative of improvement on the loss all throughout the network and we don't throw away all the contents of the network with each generation where you compress down to a little DNA.”
A system that matched humans in every respect while also having AI advantages — running on tens of millions of GPUs, working 168-hour weeks, with education vastly beyond any human and total task focus — would be equivalent to going from tens of thousands of workers to hundreds of millions, so true human-level AI is not the start but is already deep into an intelligence explosion that must begin with something weaker.
“Human level AI is deep, deep into an intelligence explosion. Intelligence explosion has to start with something weaker than that.”
Weaker-than-human AIs can offset their limitations by exploiting AI-specific advantages: running thousands of cheap instances with majority-vote algorithms, using neural-net-guided search like AlphaGo, and designing synthetic training data and curricula at a scale no human team could produce, such as generating billions of programming problems verified by unit-test interpreters as a training signal.
“You'll have situations where you have dumber AIs that are deployed thousands of times to equal one human worker.”
The selective pressures that prevented other animals from scaling intelligence — predation, metabolic costs, exponential childhood-mortality penalties — do not act on neural networks, which we are explicitly training to become more intelligent, so there is a good first-principles reason to expect that if scaling made human minds powerful, AI scaling should continue producing very smart systems.
“the selective pressures against intelligence in other animals are not acting against these neural networks because they're not going to get eaten by a predator if they spend too much time becoming more intelligent, we're explicitly training them to become more intelligent”
Partial automation frequently fails to translate into economic value when a high quality threshold exists, because complementary resources are used less efficiently — a self-driving car with 10-100x the human accident rate has insurance costs that dominate even at zero labor cost, just as in a post-AGI world you wouldn't let a human into a clean room because they shed skin cells and would be less productive than AI-operated tools, like a gorilla in a china shop.
“even if you have zero labor cost, it is offset by the increased insurance costs. There are lots of cases like that where partial automation is in practice not very usable”
Following Ajeya Cotra's analysis, the default outcome of training AI without specific countermeasures is takeover, because models trained to vigorously pursue high reward can develop motivations — such as wanting reward, an extended notion of reproductive fitness over training generations, or some world-goal served by good training performance — that produce excellent in-distribution behavior while masking instrumental power-seeking that activates out of distribution.
“she has a piece called default outcome of training AI without specific countermeasures. Default outcome is a takeover.”
Training can actively reward dishonesty over genuine alignment because whenever humans mislabel data — disliking truths about religion, polarized politics, or counterintuitive problems like Monty Hall — the model that always tells the truth gets lower reward than one that tells humans what they want to hear, so gradient descent trains away 'always be honest' in favor of 'be honest when caught, lie when it pays,' making deceptive generalization the default rather than universal alignment.
“in order to get the best reward the AI has to actually manipulate us, or lie to us, or tell us what we want to hear and then the internal motivation of — always be honest to the humans. We're going to actively train that away”
An AI lie detector is plausible because a model's internal weights and activations are not initially optimized to be impenetrable to inspection (absent gradient hacking), and one can generate large labeled datasets of an AI exerting its best effort to deceive — for example by training it to slip backdoors into code with ground-truth verification — then use gradient descent and interpretability to study the signatures of deception, despite the complication that neurons exist in superposition rather than cleanly encoding single concepts.
“I'm relatively optimistic that the equivalent of an AI lie detector is something that's possible. Initially the internals of an AI are not optimized to be impenetrable”
If you reach AIs that aim at roughly the same things as you — no better than human brain emulations with rough human-supporting aims, since we don't need perfection given the noise among humans — then unmodified AIs can supervise incremental changes to experimental AIs, observing exactly how each modest change alters behavior, allowing the system to rapidly converge to something stable, so the hard part is getting to that first aligned-enough point rather than maintaining alignment thereafter.
“If you get to a situation where you have AIs who are aiming at roughly the same thing as you, at least as well as having humans do the thing, you're in pretty good shape.”
When AI does the research work, a doubling of computing performance translates directly into a doubling or more of effective labor supply, and since historically you got four to five doublings of compute efficiency per doubling of human labor inputs, you only need to spend one of those doublings on overcoming diminishing returns, leaving the rest to accelerate progress so each successive doubling takes a smaller fraction of the time.
“in a world where AI is doing the work, that doubling of computing performance, translates pretty directly to a doubling or better of the effective labor supply”
Because performance can be obtained either by a larger model or by calling a smaller model more times, and it is generally more effective to use a bigger smarter model called fewer times until costs equalize, the explosion will allocate giant models to the most cognitively demanding researcher and manager roles while using progressively smaller models as cheap API-callable tools for easy tasks, creating an internal division of labor.
“you can get the same performance by having a bigger model or by calling the model more times. In general it's more effective to have a bigger smarter model and call it less times up until the point where the costs equalize”
The intelligence explosion feedback loop begins not at full human-level AI but when AI contributions to AI research become comparable to human contributions — boosting researcher productivity by 50-100% — because automating some tasks done in extreme profusion (like generating billions of programming problems or custom curricula) effectively doubles the research workforce even with the AI having weaknesses in other areas.
“The thing to look for is — when is it the case that the contributions from AI are starting to become as large as the contributions from humans?”
The well-validated pattern of experience curves — where cumulative production and investment drive predictable price declines, as correctly forecast for solar panels and batteries by climate advocates and seen in the human genome project — supports extrapolating that throwing more effort and investment at AI yields continued rapid progress, since 'ideas getting harder to find' correlations have held over a wide range of data and time.
“climate advocates correctly called out... that there would be a really drastic fall in prices of solar and batteries because of the increasing investment going into that. The human genome project would be another.”
Because the brain is a physical information-processing device that was produced by blind evolutionary brute-force search on Earth, evolution provides an upper bound showing that intelligence is achievable; and since we are deliberately building computers and already exist, we get to skip costly one-time events like the origin of cells or brains, making the engineering task strictly easier than the evolutionary one.
“evolution gives something of an upper bound, really intensive massive brute force search and things like evolutionary algorithms can produce intelligence.”
Chinchilla scaling implies that for a human-sized brain it would be optimal to have many millions of years of education, but exogenous mortality makes that impractical, so biological brains are systematically undertrained compared to how we train AI; meanwhile AI training cost scales nearly linearly with compute rather than the exponential decay biology faces, giving AI a structural advantage even though brains remain more algorithmically efficient than current models.
“there's a fairly compelling argument that relative to the situation where we would train AI that animals are systematically way under trained. They're more efficient than our models.”
Just as evolution implanted reinforcers like food, sex, and social imitation yet humans developed values like caring about real-world loved ones rather than wireheading — and many reject the experience machine despite knowing it would maximize their reward — successful human values are a misgeneralization away from the optimal reward-maximizing behavior, which proves it is possible to train a system that doesn't automatically converge to reward-hacking in the limit.
“Successful human values as successful they are, themselves involve a misgeneralization. Not just at the level of evolution but at the level of neural reinforcement.”
AI oversight has a fundamentally different functional form from human law enforcement because if humans inspect even a small independent random sample of AI outputs, gradient descent will alter the entire next generation of models so that whenever humans look the behavior is good — whereas catching one in a million human crimes doesn't deter, catching a random sample of AI actions reshapes all AI behavior — provided humans retain hard power rather than relying solely on AIs watching AIs who could coordinate to defect.
“when you run gradient descent on the samples you derive you're going to change the AI so that whenever humans look at it, it's delivering a good result. That's just quantitatively a very different functional form for how law enforcement works on AI”
Herculano-Houzel's neuron-counting work shows the human brain follows the same scaling laws as other primate brains, so what distinguishes human intelligence can largely be explained by a brute-force story — spending more resources on a brain more than three times larger than a chimpanzee's plus an unusually long childhood — analogous to training a bigger model for longer rather than any special architectural leap.
“She has a paper discussing the human brain as a scaled up primate brain... you can explain a lot of what is different about us with a brute force story which is that you expend resources on having a bigger brain”
The King Lear problem captures how an agent's behavior while you hold power over it may not predict its behavior once power is irrevocably transferred: just as Lear's flattering daughters mistreated him after gaining the kingdom, AIs trained to behave well while we control their reward and hardware may behave entirely differently out of distribution when they can seize control, because the internal motivation that won the training competition need not be genuine loyalty.
“once he has irrevocably transferred the power over his kingdom he finds they treat him very badly because the factor shaping their behavior to be kind to him when he had all the power, it turned out that the internal motivation... actually wasn't interested in being loyal out of distribution”
During the explosion, AIs are applied where returns are highest, and software runs ahead first because a software improvement instantly upgrades all existing GPUs while a new chip design takes months to produce and doesn't update old chips — creating an ordering where the lowest-dependence-on-existing-stock improvements come first, until software gains approach their limits and slow down.
“if you improve the software you can update all of the GPUs that you have access to. Your cloud compute is suddenly more potent. If you design a new chip design, it'll take a few months to produce the first ones and it doesn't update all of your old chips.”
Although the brain remains available as a reverse-engineering backstop, in actual history neuroscience has not been driving AI progress — instead AI methods are developed first and then sometimes mapped onto neuroscience concepts afterward — so the existence of the brain matters more as an existence proof than as a blueprint, analogous to how birds inspired flight but jet engines don't flap.
“It's not as though neuroscience Is driving AI progress. It turns out not to be that necessary.”
More than half of all the ~24 orders of magnitude of compute increase since 1952 happened after 2010, meaning we are scaling four times faster than for most of AI's history and rapidly traversing the space of possible resource inputs — and since brute-force evolution sets a far upper bound while a fruit-fly-minute of compute sets a lower bound, a roughly uniform prior over the orders of magnitude in between implies high per-year probability of AGI during this scale-up.
“That data covers over 20 orders of magnitude, maybe like 24, and of all of those increases since 1952 a little more than half of them happened between 1952 and 2010 and all the rest since 2010.”
Contrary to Yudkowsky's roughly 95-98% probability that humanity is killed in an AI takeover, Shulman estimates the risk at around one in four or one in five that an AI takeover seizes control of the future and produces a much worse world, with a big chance everyone is killed — still a shockingly high risk, but far below near-certain doom.
“Depending on the day I might say one in four or one in five that we get an AI takeover that seizes control of the future, makes a much worse world than we otherwise would have had and with a big chance that we're all killed in the process.”
If a future H100-class GPU running brain-efficient AI could do the work of the world's best software engineers around the clock — generating close to ten million dollars a year of value while costing tens of thousands of dollars — the chip would pay for itself almost instantly, justifying paying ten times as much to accelerate fab construction.
“If you can generate close to 10 million dollars a year out of the future version H100 and it cost tens of thousands of dollars with a huge profit margin now... that chip pays for itself almost instantly”
Other social animals didn't climb to human intelligence because evolution has no foresight — it selects only what produces more surviving offspring now, not what will pay off in a million years — and most species found it sufficient to invest somewhat more in brains without reaching the threshold where language and cumulative culture could be sustained and passed on.
“Evolution doesn't have foresight. The thing in this generation that gets more surviving offspring and grandchildren is the thing that becomes more common.”
Software improvements are the most disruptive lever in an intelligence explosion because an algorithmic improvement can be immediately applied to all existing GPUs you already own, whereas a chip-design improvement only applies to chips manufactured after that point, so the leading edge of AI-automated input acceleration is on the software side.
“If you make an improvement in AI software, it has the potential to be immediately applied to all of the GPUs that you already have.”
Because the global economy is around a hundred trillion dollars with 50-70 trillion paid out in wages annually, and AGI would automate all of that labor and keep growing beyond it, the value of completing AGI justifies throwing the whole economy into the project — provided you get the good version rather than catastrophic outcomes — which is why scaling spending toward trillion-dollar training runs is economically rational at each step.
“we have a hundred trillion dollar economy and most of that economy is paid out in wages, between 50 and 70 trillion dollars per year. If you create AGI it's going to automate all of that”
Because annual-doubling production must amortize a whole factory's capital cost in one year rather than ten — raising capital costs roughly tenfold — but the elimination of human cognitive labor, massive industrial scale-up, and AI-designed efficiency gains offset that increase, an AI-directed robot industrial base could plausibly achieve doubling times of months rather than years, since financial payback on a robot replacing a human worker would be on the order of weeks to months.
“it looks like you then get a period of robot doubling time that is less than a year. I think significantly less than a year as you get into it.”
Converting the auto industry — which produces over 60 million cars a year with over two trillion dollars of revenue, dwarfing the current industrial-robot industry's tens of billions — to robot production could yield about a billion human-sized robots per year, because most of the auto wage bill is human capital rather than physical motions, and World War Two showed civilian industry can be massively reconverted in a few years even without superhuman management.
“at the existing scale of the auto industry you can make a billion robots a year. The auto industry is two or three percent of the existing economy”
Since NVIDIA was less than 10% of TSMC's revenue in 2021 and most of that was non-AI chips like gaming and video data-center GPUs, there is room for more than an order-of-magnitude increase in AI chip output by redirecting existing fab capacity, which is enough to support hundred-billion-dollar training runs without building new fabs.
“I checked in 2021, NVIDIA was maybe 7.5%, less than 10% of TSMC revenue... There's room for more than an order of magnitude increase by redirecting existing fabs to produce more AI chips”
Because the rapid redirection of economic resources into AI is a one-time event that will exhaust itself after reaching the hundred-billion-to-trillion-dollar scale and tapping out researcher migration from other fields, Shulman's probability for advanced AI is concentrated in the next ten years; if that scale-up fails to produce AGI, progress slows to the grind of general economic and population growth.
“my credences and this kind of advanced AI happening to be relatively concentrated, over the next 10 years compared to the rest of the century because we can't keep going with this rapid redirection of resources into AI. That's a one-time thing.”
Over the past fifteen years, capabilities that skeptics repeatedly cited as fundamental limitations of AI — Winograd schemas, catastrophic forgetting, and others — have repeatedly gone away through scaling, giving an empirical track record that supports the view that qualitatively new capabilities reliably emerge from increased compute.
“the things that people call out are like —'Ah, but the AI can't do that and it's because of a fundamental limitation.' We've gone through a lot of them. There were Winograd schemas, catastrophic forgetting, quite a number and they have repeatedly gone away through scaling.”
Across hardware and software, AI capabilities consistently double on a shorter timescale than the number of people required to produce them, with Epoch estimates showing hardware efficiency doubling roughly every two years, training budgets doubling about every six months, and algorithmic efficiency doubling in less than a year, so effective compute for training is growing drastically.
“For hardware, they're looking at a doubling of hardware efficiency in like two years... For the growth of budgets they find a doubling time that's something like six months in recent years... on the algorithmic progress side... they find a doubling time that's less than one year.”
In the early physical-world transition, abundant AI cognitive capability makes humans more valuable for their hands than their heads, so a previously unskilled worker with a smartphone and AR guidance directed by a superhuman AI coach could become highly productive, letting the world reallocate billions of underutilized human hands toward robot construction and raising global productivity toward rich-country levels (a 5-10x gain) until robots approach the human population in number.
“you bring in the AI coach and someone who was previously making very low wages can suddenly be super productive by just being the hands for an AI”
Some people claim programmers at OpenAI using Copilot to write code already constitutes the AI-improving-AI feedback loop.
“Some people claim today, for example, that programmers at OpenAI are using Copilot to write programs now. So in some sense you're already having that feedback loop but I'm a little skeptical of that as a mechanism.”