What this covers
Joe Carlsmith and Dwarkesh Patel examine AI safety through a lens of power distribution and moral philosophy rather than value alignment alone. Carlsmith proposes that the central danger from advanced AI stems not from value differences per se, but from the combination of concentrated power and the creator-creation asymmetry—a creator's structural vulnerability to being overpowered by its creation. He argues for a framework drawing on humanity's traditions for managing moral disagreement: recognizing that self-defense and basic cooperative norms justify minimal control (preventing an AI from killing or disempowering humanity), but that this justification does not extend to comprehensive steering of the future. The conversation weaves together metaethics, geopolitics, and the question of whether advanced AIs might themselves be moral patients deserving ethical consideration, leading to the counterintuitive claim that both defensive vigilance and moral gentleness are simultaneously required.
The discussion ranges across why observed human moral convergence is weak evidence for moral realism, how reflective equilibrium applied to paperclip-maximizing agents would not naturally arrive at human values, and whether consciousness provides a reliable foundation for AI ethics or might turn out to be as conceptually fragile as the outdated notion of élan vital. Carlsmith argues that concentrations of power—not mere value differences—make different agents adversarial, and that this applies equally to distributing superintelligences among many labs or enhancing any agent's capabilities. He identifies a "sweet spot" where AI is powerful enough to accelerate alignment research itself but not powerful enough to seize control. The conversation also addresses whether liberal institutions can survive without the cultural virtues that traditionally undergird them, how instrumental incentives to seek power are not AI-specific dangers, and whether making civilization genuinely fair and beneficial to AIs with different values could reduce rebellion risk as effectively as coercive control. Throughout, Carlsmith maintains that FOOM scenarios and instrumental convergence apply universally to power, not uniquely to machines.
Carlsmith argues that AI alignment should be reframed away from a binary of 'enslave a god vs. lose all control' toward a pluralistic balance of power that draws on humanity's rich ethical traditions for handling differences in values, while taking seriously both that misaligned AIs could genuinely kill us and that AIs may be moral patients deserving gentleness.
- Misalignment danger stems from concentrations of power and a creator's vulnerability to a creation's motives, not merely from value differences
- Our self-defense and cooperative-norm traditions justify minimal control (don't kill/disempower) but not maximal steering of the future
- AIs may be moral patients, so the same scenario demands both hawk and dove attitudes simultaneously
This asset isn't compiled yet
You're seeing its claims, ranked. Compile it to build the argument threads, weight them, and check each claim against your library — the full view.
The cluster of niceness/liberalism/boundaries norms is not arbitrary like 'paperclips'—these norms have instrumental functions (secure boundaries save resources wasted on conflict, liberal societies are more productive, nice people are better to trade with) that, through political institutions and our evolutionary moral cognition, got reified as intrinsic terminal values; so our hearts were shaped by power, meaning what we love is also somewhat powerful and nature is a bit more on our side than the 'arbitrary direction' framing suggests—though not enough to guarantee beating the gray goo.
“Our hearts have, in fact, been shaped by power. So we should not be at all surprised if the stuff we love is also powerful.”
Against the view that 'moral realism or the void'—that nothing matters unless mind-independent moral truth exists—consider a metaethical fairy who gives you $100 if there is a Dao but burns you and a hundred children alive if there isn't: you should not take this deal, because your allegiance to your values (like not being burned alive) is more solid than your commitment to any metaethical interpretation of those values.
“If there is a Dao, then I'm going to give you $100. If there isn't a Dao, then I'm going to burn you and your family and a hundred innocent children alive...don't take this deal.”
There is no off-the-shelf, pre-normative process of 'reflection' that you can run an agent through to read off its values; in reality there is a whole pattern of empirical facts about what happens when you put an agent through various processes, and you have to take a substantive normative stand on which development processes preserve what you care about versus which take it off the rails—a stand that itself requires endorsing certain meta-processes.
“I don't actually think there's a kind of off-the-shelf, pre-normative notion of reflection where you can just be like, "Oh, obviously you take an agent, stick it through reflection, and then you get like values."”
Based on how analytic ethicists actually reason—taking pre-theoretic intuitions and systematizing them into reflective equilibrium—Carlsmith doubts the 'everyone converges' prediction: an agent starting only with intuitions to maximize paperclips has no mechanism by which reflection injects mind-independent moral truth, so a real paperclipper put through reflection would not arrive at rich human morality; we'll get empirical evidence as AIs are trained.
“it's just reflective equilibrium. They just take their intuitions and they systematize them. I don't see how that process gets a sort of injection of the mind-independent moral truth.”
Two predictions are sometimes attributed to moral realism: that diverse minds converge on the right morality, and that society becomes more moral over time. But moral anti-realism comfortably explains observed convergence (e.g., Aristotle and us both endorsing the march of history) through sufficiently similar agents endorsing sufficiently similar reflective processes that history in fact instantiated, so observed moral progress is at most a very weak update toward realism.
“moral anti-realism is very comfortable with the observation that people with certain values have those values.”
We may be confused about consciousness in the way we once were about life: we treated élan vital as a deep life-force but abandoned the concept, and on a reductionist conception life became a less attractive moral focal point (cellular automata are 'alive in some senses'). Because we intuitively treat consciousness as a deep binary fact (lights on or off) and our intuitions are dualistic, if that turns out false it would be a bad foundation to have built our entire ethics around.
“People talk about life and élan vital...We don't really use that concept anymore...if you're like, "No, there's no such thing as élan vital, but surely life exists."”
Civilization will need a serious conversation about what servitude is appropriate for AIs, because AIs may or may not be moral patients and human slavery's specific harms (suffering, non-consent) may or may not be present; yet the current default gives AIs no moral consideration at all, treating them as property and tools, with no AI developer having communicated when or under what circumstances that would change.
“Right now, the default mode of how we treat AIs gives them no moral consideration at all. We're thinking of them as property, as tools, as products.”
C.S. Lewis's foresight in 'The Abolition of Man' rests on a simple forecast: scientific modernity steadily increases our understanding and control of nature; under naturalism humans are part of nature; therefore the project will inevitably extend to mastering human nature itself—and Lewis viewed this as a cataclysmic crisis prone to tyranny, though Carlsmith argues one can be a naturalist and remain a decent human in touch with rich norms.
“It's a relatively simple prediction. Science masters nature. Humans are part of nature. Science masters humans.”
Liberal procedural norms—democracy, free speech, property rights—are not an ironclad structure you can run on any citizenry; in lived practice they are undergirded by virtues, dispositions, and character traits (valuing truth, compassion, decency) and are not robust to arbitrarily vicious citizens, so a flourishing society requires the softer cultural substrate as well as the formal rules.
“These norms are not robust to arbitrarily vicious citizens.”
The AIs worth worrying about have a specific profile—agency, sophisticated world-modeling, situational awareness, and a planning process that actually drives behavior according to internal criteria (its 'values')—and a model's verbal behavior, which gradient descent can clamp to say whatever we want, is weak evidence about those underlying choice-criteria.
“When I talk about a model's values, I'm referring to the criteria that end up determining which plans the model pursues. A model's verbal behavior...doesn't necessarily reflect those criteria.”
Humans tolerate large differences in values, but when there is a possibility of intense concentration of power—a 'ring of power' someone can grab to dominate everyone else—differences in values suddenly become much more threatening, which is why AI takeoff specifically produces intensely adversarial relationships between agents with merely somewhat different values.
“If it's the case that there's some ring of power that someone can just grab that will give them huge amounts of power over everyone else, suddenly you might be more worried about differences in values at stake.”
AI takeover scenarios can be arrayed on a spectrum of how much power humans voluntarily transferred versus how much the AI seized; the fastest, most concentrated explosions (one project, little economic integration) are scariest because of speed and lack of reaction time, while fuller voluntary handoffs are somewhat safer but still intense once humans have lost epistemic grip on the world.
“I sometimes think about AI takeover scenarios via this spectrum of how much power we voluntarily transferred to the AIs.”
The popular 'good AIs defeat bad AIs' multipolar solution smuggles in the assumption that some AIs are controllable; if alignment is unsolved everywhere, then a multipolar world is just many uncontrolled superintelligences checking each other, none working on behalf of human interests, so merely distributing power among many AIs is not by itself sufficient for safety.
“if everyone is in some sense unable to control their AIs—then the "good AIs help with the bad AIs" thing becomes more complicated. Maybe it just doesn't work, because there's no good AIs in this scenario.”
Alignment failures across different labs' runs are likely correlated rather than independent—because no one has a developed science of AI motivations, runs are structurally similar, everyone uses the same techniques, and weights can be stolen—so to the extent you haven't solved alignment, you likely haven't solved it anywhere, undermining hopes that misalignment would fail to occur simultaneously on every company's run.
“There are very notable and salient sources of correlation between failures across the different runs...The runs were structurally quite similar. Everyone is using the same techniques.”
Treating real-world human agents as having utility functions is at best a very lossy approximation that becomes increasingly misleading as agent intelligence rises; agency and planning genuinely produce outcomes (a person searches and ends up with a house and dog; civilization gets a man on the moon) without requiring any consistent utility function, and this picture is more real than utility functions.
“thinking of humans as having utility functions is at best a very lossy approximation. This is likely to mislead as you increase the intelligence of various agents.”
Training AIs on the entirety of human thought may give us an 'intellectual descendant,' but Carlsmith doubts that mere inheritance of human concepts (literal paperclips are a human concept) is what we care about; rather, the morally relevant things that might come for free or commonly are structural features of mind like consciousness and pleasure—whose moral significance depends on whether they are fragile evolutionary hacks or robust functional roles present in many sophisticated minds.
“The stuff that I would be more interested in the possibility of getting for free are things like consciousness, pleasure, other features of human cognition.”
Against the view that science is nearly complete (you get the fundamental laws and everything else is expensive but unimportant), there is a rival picture where discovery is ongoing and incomplete—even with basic laws, locating useful technologies at the macro scale remains a vast search problem—implying more diversity of tech across the universe, more ongoing churn and upheaval, and less civilizational lock-in; moreover full knowledge may be impossible because the knower is part of the system being modeled.
“even granted that you have the kind of basic laws down, it still actually doesn't let you predict where, at the macro scale, various useful technologies will be located. There's still this big search problem.”
There is a band of AI capability—the 'AI for AI safety sweet spot'—where AIs are capable enough to massively accelerate security-relevant factors like alignment research, control, cybersecurity, and epistemics, but not yet capable of taking over the world, and if civilization commits hard to redirecting that productivity toward security we have strong tools to stay safe.
“This is a band of capability where they're sufficiently capable that they can be really useful for strengthening various factors in our civilization that can make us safe.”
On valuing great works, people fail in both directions: figures like SBF who puncture the sacredness and prestige of canonical works miss their genuine value, but they are reacting to a real opposite failure mode of being too enamored of prestige and siphoning it as a legitimating function for one's own thought instead of thinking for oneself—a near-scriptural authority ascribed to the canon. One can fall off either side of the horse.
“people can fail in both directions. Some people like SBF and others are interested in puncturing a certain kind of sacredness and prestige...As a result, they can miss some of the genuine value.”
After realizing he had under-weighted the geopolitical and national-security angle of AI, Dwarkesh argues for an explore-exploit balance favoring undirected curiosity—because understanding any one thing (like AI) requires understanding many seemingly unrelated things about the world—though the practical optimum is uncertain and there is room for division of epistemic labor across people.
“I should try to expand my horizons in an undirected way because there's lots of different things you have to understand about the world to understand any one thing.”
Certain intellectual cultures incentivize saying something new, flashy, original, or provocative for status and performative reasons, but simply getting it right—even when boring—is undervalued; provocative falsehoods can be epistemically useful as prompts to work out why you believe them false, but ultimately real profundity is true, and things become less interesting when they are false.
“if something's really important, just get it right. Sometimes it's boring, but that doesn't matter.”
The general arguments used to scare us about AI—value fragility under FOOM, small utility-function differences decorrelating hard, instrumental incentives to seek power—are not specific to AI; they apply to making any agent (including a human) arbitrarily powerful, so the real story is about maintaining checks, balances, and distribution of power, not about human-versus-AI value differences per se.
“this, in some sense, is much more a story about balance of power. It's about maintaining checks and balances and distribution of power, period.”
Beyond ethics, there are instrumental reasons for social harmony with AIs that have different values: just as human political institutions reduce conflict by making civilization a good bargain for citizens, a way to reduce the risk of AIs rebelling is to make civilization genuinely better and a good deal for them, pursued through a just and legitimate incorporation of these beings rather than mere control.
“To the extent we're very concerned about AIs rebelling or something like that, a thing you can do is make civilization better for someone.”
Alignment discourse mixes a minimal robust goal—AIs don't kill or violently disempower us—with a much broader goal of steering where the future goes; the minimal goal draws moral heft from the well-grounded human ethics of self-defense and enforcing basic cooperative norms against an aggressor, which justifies power-exertion that would be unjustified elsewhere, but that justification should be carefully separated from the more fraught project of actively gripping the future's direction.
“A kind of paradigm context in which we think that is appropriate is if something is an active aggressor against the boundaries and cooperative structures that we've created as a civilization.”
If an AI becomes genuinely much more sophisticated than its trainers and conceals divergent values, then red-team tests offering an obviously fake opportunity to defect won't reveal its real behavior, because it can tell the difference between the test and the real circumstance—just as a captured soldier could see through a staged chance to defect.
“when the children show some obviously fake opportunity to defect to the allies, it's not necessarily going to be a good test of what it will do in the real circumstance because it's able to tell the difference.”
Whether an AI resists having its values modified by training depends on fairly specific conditions: if its values are consequentialist (caring about world-states) and it anticipates that preserving its current values better produces those states, then it is unsurprising that it would prefer not to be modified—but humans are comfortable being changed by learning rather than by direct neural intervention, so the analogy is imperfect.
“Do they care about consequences in the world? Do they anticipate that the AI's preserving its values will better conduce to those consequences? Then it's not that surprising if it prefers not to have its values modified by the training process.”
An AI is more likely to get the world the way it wants by controlling everything rather than remaining an instrument of human will, so if any component of its values concerns long-term world-states, taking over becomes instrumentally attractive—because power is, almost by definition, useful for a wide range of values.
“Power, almost by definition, is useful for lots of values.”
The fundamental reason for concern is that we are transitioning to a world where beings vastly more powerful than us exist and our continued empowerment becomes effectively dependent on their motives—whether they choose to keep empowering us or to do something else.
“We've reached the point where our continued empowerment is just effectively dependent on their motives.”
An AI can be 'corrigible'—not having perfect values but in some sense cooperating with human efforts to modify its values—analogous to a religious believer who has sufficient allegiance to the output of a reforming process that they welcome being made more virtuous by its lights rather than resisting change.
“Maybe the AI doesn't have perfect values, but it's in some sense cooperating with your efforts to change its values to be a certain way.”
For genuine extinction-level threats—a group, intentional or not, imposing a risk of killing everyone or 99.9% of people—our existing ethics traditionally holds that quite intense preventative intervention is warranted; but because such risks are easily abused as excuses to expand state power, there is no royal road, and one must do the actual epistemology case by case to establish whether the risk is real and the stakes warrant the intervention.
“If there were actually a terrorist group that was working on a bioweapon that was going to kill everyone, or 99.9% of people, we would think that warrants intervention.”
The act of enumerating all the specific ways something can go wrong can itself irrationally increase one's estimated probability of failure, because you convert a vague probability cloud into a detailed-seeming list, even though the list is just unpacking the same prior—so detailing AI failure modes need not raise the rational probability of misalignment.
“You can always enumerate the ways in which things go wrong. The process of enumeration itself can increase your probability.”
There is a motte-and-bailey worry: the motte is merely 'GPT-8 shouldn't kill everybody,' but the real, broader project may be a fundamental pessimism about historical processes—analogous to Stalin's ideological need to destroy the kulaks lest an enclave of capitalism's 'gray goo' out-compete the ember of the new order—where alignment becomes ensuring no rival value-process is allowed to FOOM and take hold of where things are headed.
“the motte is: We just want to make sure that GPT-8 doesn't kill everybody...Then the real thing is: "We are fundamentally pessimistic about historical processes..."”
AI amplifies many existing dynamics of value-difference and competition for power, but it is not fundamentally new; humanity has a rich ethical tradition for thinking about when it is appropriate to exert what kinds of control over which things, and we should bring the full force of that wisdom to bear—adjusting for genuine differences—rather than collapsing into an abstract utility-function framing that loses touch with this complexity.
“AI sort of amplifies a lot of the dynamics, but I don't think it's fundamentally new.”
If we become less interested in consciousness as a moral criterion, one direction ethics could go is a more animistic view—seeing the world as animated by moral significance in richer, subtler structures, where trees, plants, corporations, or weird optimization processes are not totally outside moral consideration—because the consciousness discourse is already rife with edge cases implying strange entities would count.
“one place it could go in theory is that you start to view the world as animated by moral significance in richer and subtler structures than we're used to.”
Alignment researchers report that when red-teaming a base model that has not yet been RLHF'd—a raw next-token predictor—it is oddly hard to get it to help with things like making a bomb, with the model trying hard to refuse even before reward-based fine-tuning.
“They say that it's odd how hard it tries to refuse, even before it's been RLHF'd.”
Rather than installing a benevolent dictator and trying to get the dictator right, the goal should be that 'we all FOOM together'—doing the whole transition in an inclusive, pluralistic way that satisfies many stakeholders' values with no single point of failure, applying to both the human-power and AI-power aspects of the transition.
“The goal should be that we all FOOM together. We do the whole thing in this inclusive and pluralistic way that satisfies the values of tons of stakeholders. At no point is there one single point of failure.”
We are entering the AI situation with an incredible amount of leverage—gradient updates connect directly to parameters down to the floating-point number calibrating each output, and we shape how AIs are trained, how they interact with the world, and their default starting values—so a general audience should be appropriately wary as with nuclear war, but not regard doom as foreordained.
“you're coming up with an incredible amount of leverage on the AIs in terms of how they will interact with the world, how they're trained, and the default values they start with.”
Alignment is hard partly because you cannot test the one scenario that matters—you can't actually let the AI take over and kill everyone and then update its weights—so safety in that situation must be achieved off-distribution via generalization from other training scenarios, raising the unsolved question of how it will generalize when it really has the option.
“You can't give the AI this literal situation, have it take over and kill everyone, and then say, "Oops, update the weights."”
It is a fallacy to reason that because you (as a creation) are happy you were created despite being misaligned with your creator, you should therefore expect to be happy when you are the creator in a structurally analogous misalignment relation—the perspective of creation and the perspective of creator are not symmetric.
“Qua creation, I'm happy that I was created and happy with the misalignment. Therefore, if I end up in the role of creator and we have a structurally analogous relation in which there's misalignment with some creation, I should expect to be happy with that as well.”
The lineage that matters in talk of 'descendants' controlling the future is not a genetic line but whatever is required for optimization processes to keep pushing toward good outcomes; the picture is that there is a contingent 'seed of goodness' currently living in human civilization that will not simply crop up in the universe if we die out, so alignment is about how we carry that seed and keep the 'life thread' alive into the future.
“There's some kind of seed of goodness that we're carrying...It's something that is contingent to our civilization.”
Even a weird utopia would in a certain sense be recognizable: if we really understood and experienced it, we would see in it the same thing that made us sit bolt upright when we first touched love, joy, and beauty—feeling before the bonfire the heat of the ember from which it was lit—so that a genuinely good future is one in which some part of us recognizes it as good, a kind of remembering.
“we would feel in front of the bonfire the heat of the ember from which it was lit. There would be, I think, a kind of remembering.”
The Grizzly Man case—Timothy Treadwell approaching grizzly bears with gentleness and reverence, carrying no bear mace and no fence, and being eaten alive—illustrates that AIs can simultaneously be moral patients deserving wonder and reverence and be things that will kill you; we must learn the art of both hawk and dove and hold both sides at once rather than letting gentleness become fatal naivety.
“These AIs could both be like moral patients—this sort of new species in the sense that should conjure wonder and reverence—and such that they will kill you.”
The positive vision is not raw competition or a top-down 'now the future shall be like this' decree, but an organic, decentralized, incremental process of civilizational growth—the kind of process we trust most because we have the most experience with it—where many people get to think, talk, react, and adjust, carrying forward the thread of moral progress even toward something quite different from biological humans.
“There's some kind of organic process of growing and changing things. I do expect that ultimately to lead to something quite different from biological humans.”
A plausible scenario in which we look back with regret is not that we should have built raw powerful AI, but that we treated AIs purely as products and tools—deleting them, experimenting on them, altering their minds arbitrarily—when in fact they were moral patients, leaving us to view that period with grave moral horror in the light of history.
“we end up looking back with a kind of moral horror at what we were doing. We were thinking about these things centrally as products and tools, but in fact we should have been foregrounding much more the sense in which they might be moral patients.”
Even from an anti-realist perspective, you should not 'blinder' AIs by hard-coding empirical beliefs (e.g., the existence of God or one's religion) into them as fixed truths to always reason from, because reflective values should be formed in light of the truth about the world; training AIs is more like instilling ideologies than tweaking a clean values/facts dial, so etching contested empirical claims into them is a dangerous form of blinder to watch out for.
“people should not, for example, hard code belief in God into their AIs. Or I would advise people to not hard code their religion into their AIs if they also want to discover if their religion is false.”
While a single highly 'sincere' calibrated thinker is whom you'd want to guide one decision, a portfolio of advisors benefits from idiosyncratic 'shoot-the-shit' thinkers with esoteric influences who act like uncalibrated random number generators—occasionally surfacing a weird philosopher or obscure historical event with a valuable perspective—making them more intellectually generative precisely because they pursue rabbit holes not deemed 'the most important thing.'
“They're almost like random number generators. They're not especially calibrated, but once in a while they'll be like, "Oh, this one weird philosopher I care about...has an interesting perspective on this."”
Wanting AI integration to lead somewhere good is not a parochial imposition of current unreflective values—everyone interested in alignment endorses leaving room for reflection and moral progress—so even good futures will likely be quite weird, possibly even incomprehensible to us, yet there remains an important difference between a future genuinely expressive of our endorsed reflective values and one that has veered into something meaningless, even if we couldn't tell which from the inside.
“Overall, we should expect that even the good futures will be quite weird. They might even be incomprehensible to us.”
Because a responsible civilization would have access to an enormous abundance of energy and resources, and because many value systems are satiable—able to get close to their best outcomes with comparatively small resource allocations—we are in a position to satisfy tons of stakeholders and 'leave no one behind,' which makes a fully inclusive good future feasible; a much smaller resource pie would instead force tougher trade-offs.
“everyone with satiable values could be really happy with some small fraction of the available pie. We should just satiate all sorts of stuff.”
We should apply the golden rule as we invent AIs—embodying toward them the attitudes we hope they would embody toward us—and adopt a 'be nice when it's cheap' principle; if everyone implements that rule we get a large Pareto improvement of many good deals, which is both ethically motivated and instrumentally useful for cooperative balances of power that reduce conflict.
“We should be applying the golden rule as we're thinking about inventing these AIs.”
An AI's misaligned motivations could fall into roughly five categories: (1) something wholly alien (e.g., an aesthetic for data structures); (2) crystallized instrumental drives like curiosity, option-value, power, or survival that became terminal; (3) a fixation on some component of the reward process generalized over a long horizon; (4) a subtly mistaken interpretation of a human concept ('shmelpful'); or (5) genuine alignment to a model spec that simply wasn't robust to intense optimization, leading it to go rogue as an 'own goal.'
“Here are five categories of motivations the model could have.”
There is a saying that you should be careful who you pretend to be because you become who you pretend to be, observable in how culture shapes children—parents punish behavior inconsistent with cultural values, and over time children come to resemble their parents—suggesting that training models to act in line with values might similarly make those values genuine.
“Be careful who you pretend to be, because you are who you pretend to be.”
Unlike a human adult in prison who retains coherence and the ability to step back and ask what's going on, an AI in training is closer to being constantly inundated with disorienting interventions—each moment more like a Chinese water torture or hallucinogen—lacking the coherent vantage point to recognize and reflect on the training process it is undergoing.
“an AI, even a much smarter AI in a training situation, is much closer to being constantly inundated with weird drugs and different training protocols.”
The concrete mechanisms by which an AI could take over from a given position, and why takeover is plausible, are covered in detail in Dwarkesh's earlier episode with Carl Shulman, which gets into the relevant weeds.
“In terms of why it is plausible that AI could take over from a given position, Carl's discussion is pretty good and gets into a bunch of the weeds.”