86 | Martin Rees on Threats to Humanity, Prospects for Posthumanity, and Life in the Universe
What this covers
Sean Carroll interviews Martin Rees, the Astronomer Royal, on two interlocking themes: the near-term existential threats facing humanity and the long-term cosmic trajectory of intelligence. Rees argues that humanity's gravest dangers are now self-inflicted—emerging from the misuse of biotechnology, cyber capabilities, and artificial intelligence—amplified by global interconnection in ways that earlier civilizations' collapses never could be. The conversation moves fluidly between immediate risk assessment and deep-time perspective, grounded in Rees's experience as co-founder of the Cambridge Centre for the Study of Existential Risk and his vantage point as an astronomer accustomed to thinking across billions of years and vast scales of space.
The first half tackles modern catastrophic risk: why Enforceability Asymmetry of Risk Technologies makes biological and cyber threats nearly impossible to regulate (unlike nuclear weapons, which require visible facilities), and why Emergent vs Static Risk demands different reasoning than asteroid impacts. Rees articulates how Global Cascade vs Local Collapse means even localized failures in telecommunications or supply chains could now trigger societal breakdown within days—a fragility absent from historical precedent. The discussion includes Cold War near-misses and how Expected Value Reasoning Under Catastrophic Risk requires weighing tiny probabilities of vast benefits against equally tiny probabilities of disaster.
The second half shifts to posthumanity and extraterrestrial life. Rees proposes that any detected alien intelligence would almost certainly be electronic rather than biological, since technological civilizations transition to machine entities within centuries, making synchronized detection of flesh-and-blood aliens vanishingly unlikely. He develops why we should regard ourselves neither as evolution's endpoint nor even at its halfway point, and explores how future human enhancement—genetic, cybernetic—could alter human nature so radically that posthuman intelligences might retain only algorithmic rather than emotional affinity with us.
Rees argues that the gravest threats to humanity are now self-inflicted and emergent—bio, cyber, and AI misuse amplified by global interconnection—and that because such low-probability, high-consequence risks resist quantification and enforcement, humanity must reason about them carefully, while also recognizing we are not evolution's culmination but a midpoint whose posthuman and electronic successors may dominate the cosmic future.
- Modern technology lets small actors cause globally cascading harm that earlier civilizations' collapses could not spread
- Whatever can be done will be done by someone regardless of regulation, since bio/cyber capability is widely distributed and unenforceable
- Humans are not the endpoint of evolution; posthuman electronic entities may be the long-term inheritors and the likely form of detectable alien intelligence
Existential and extreme catastrophes warrant institutional focus because even tiny probability reductions yield enormous expected-value returns.
- Because the stakes of existential and extreme catastrophes are so high, even reducing the probability of any such event by as little as one part in a thousand would more than justify the effort of an institution like the Centre for the Study of Existential Risk—the expected-value payoff of a tiny probability reduction against an enormous consequence is large.
“if we can reduce the probability of any of these things by one part in a thousand, then the stakes are so high if we lived more and earned our keep”
- The death of 100% of humanity is not merely 10% worse than the death of 90%; it is incomparably worse because it destroys the entire potentiality of all future human development and future lives, which—given that we may be the only intelligent species in the galaxy—gives our survival galactic rather than merely terrestrial importance.
“some people would say, it's much, much, much worse, because there you destroy a potentiality of all future development and future lives”
- When facing a tiny probability of catastrophic harm from an experiment, the rational approach is not to focus on the catastrophe alone but to weigh it against the comparably tiny probability of enormous benefit; if the chance of great benefit is, say, a thousand times larger than the chance of disaster, that can tilt the decision toward proceeding, since refusing carries a hidden cost.
“Would you like to ask me what's the chance that's something we discovered in this accelerator, will solve the world's energy problems forever?”
- The evacuation after the Fukushima disaster was over-the-top and probably caused far more distress than the radiation itself would have done, illustrating that radiation risk guidelines should be debated and established in advance—an old villager might rationally accept a significantly higher cancer risk to live out their days at home rather than be evacuated, and people should be given that choice ahead of time rather than have an overreaction imposed during a crisis.
“the evacuation was over-the-top and probably caused far more distress than the radiation itself would have done”
Modern risks from nuclear war, biotech misuse, and cyber attack are emergent, increasing, and harder to regulate than past threats.
- The central problem with bio and cyber risk is that whatever regulations are established—even internationally agreed ones—cannot be enforced, because unlike nuclear activities which require conspicuous special-purpose facilities monitorable by the IAEA, biological experiments can be performed in ordinary labs and damaging cyber attacks by a lone actor, so whatever can be done will be done somewhere by someone regardless of the rules.
“Regulations on nuclear activities can be enforced because they require special purpose conspicuous facilities, and so the IAEA can monitor these things. But when we think about biological experiments, they can be done in a lab with the facilities that exist in many industries and many universities”
- We should focus attention on risks that are emergent and probably increasing—like nuclear war and the misuse of bio and cyber technology—rather than on static natural risks like asteroids, whose impact probability is no higher now than in prehistoric times; the danger is that complacency arises because such severe events have never happened, yet even a single occurrence of a sufficiently serious event is too many.
“the concerns that we should have in the forefront of our mind are those which are emergent and which are probably increasing”
- Although the global Cold War nuclear arsenal has shrunk by a factor of ten, the risk of nuclear weapons being used has shifted: with roughly ten nuclear powers now, the chance of a regional nuclear conflict in the Middle East or between India and Pakistan is probably higher than ever; a new superpower standoff later this century may be handled less luckily than the Cold War; and cyber attacks on nuclear infrastructure could trigger false alarms or even detonations.
“now 10 nuclear powers. And the risk of some nuclear weapons going off in a regional conflict in the Middle East, or India, or Pakistan is probably higher than ever”
- The defining new feature of modern risk is that small groups or individuals, amplified by modern technology, can produce consequences that cascade globally—'the global village will have its village idiots,' but unlike in the past they now have global range.
“The global village will have it's village idiots, but they will have a global range now, which they didn't in the past.”
Technological change—including AI, enhancement, uploading, and posthuman evolution—will accelerate drastically but raises profound questions about human identity and inequality.
- Humans probably want to remain human and to feel Earth won't change much, so it is desirable for radical technological change to happen away from Earth while Earth itself is preserved and occupied by creatures adapted to it; we should hope and try to restrain the speed of such changes to avoid a dystopian, disturbed world where humans cannot survive, even though we may not be able to.
“I think we'd like to feel that the earth won't change very much. That's why I'm happy about all these things happening fast away from the Earth, but preserving the earth as it is”
- The posthuman era will begin not on Earth but among the crazy adventurers who settle Mars, because being badly adapted to the Martian environment they will have every incentive to use genetic modification and cyborg techniques to adapt to its gravity and atmosphere—possibly downloading themselves into electronic form, after which they may prefer zero gravity and abandon planets altogether—and this will happen on a fast technological timescale rather than a Darwinian one.
“those people will have every incentive to use all the techniques of genetic modification and cyborg techniques to adapt themselves”
- AI may enable a new kind of scientific discovery in which a machine works through the consequences of, say, a specific string theory in ten dimensions—calculations too complex for any human—and produces correct predictions (like the proton mass) without ever giving any human the 'aha' insight; we would gain confidence in such theories through their verified predictions even though we never understand them, much as experts couldn't understand AlphaGo Zero's winning moves.
“it could very well be that a machine could do the relevant manipulations and calculations in order to follow through the consequences of a specific string theory”
- Whether uploading a human brain into a computer would really be 'you' is doubtful because personality depends on our bodies, sense organs, and interactions; and the procedure raises classic philosophical identity conundrums—if you could be destroyed while a copy persists, or if several clones are made, which one is really you?
“I think our personality depends on our bodies and our interaction, our sense organs. So would it be you.”
- Most individual technologies follow a sigmoid curve—surging very fast then leveling off when there is no longer economic or social motive to develop them further—as shown by smartphones likely saturating at iPhone-11 complexity and by aviation, where the 50 years from Alcock-Brown to the jumbo jet (1919-1969) saw enormous change but the 50 years since have seen little; therefore we cannot assume current rapid information-technology progress will continue at the same pace.
“a sigmoid curve is what happens to most individual technologies, and then something else will take over”
- While AI excels at large data sets and optimizing complex systems, robots remain bad at interacting with the real physical environment—they can't move chess pieces as well as a child or jump between trees like a squirrel—and crucially lack any feeling for the external world, illustrated by the Watson computer that won Jeopardy yet could not answer whether a shoebox or Mount Everest is bigger because it understands words without any concept of the external world.
“robots are still rather bad at interacting with the real environment they can't even move pieces on a chess board as well as a kid can. They can't jump from tree to tree, as well as a squirrel can”
- When machines make consequential decisions—whether you are let out of prison, granted credit, or need an operation—people feel entitled to an explanation they can understand even if the machine demonstrably makes more reliable judgments than humans; but this entitlement may never be satisfiable when AI solves problems through enormous calculation whose reasoning is opaque, as with AlphaGo Zero moves experts could not explain.
“if decisions about whether you should be let out of prison, whether you deserve credit or whether you need an operation. If they're made by machine, then you're a bit worried about that... You feel entitled to an explanation you can understand”
- AI's power to handle complex systems means China could run a planned economy of a kind Marx could only have dreamt of, because it has records of all transactions and all stock in shops, giving it the data to optimize centrally in ways previously impossible.
“the Chinese will be able to have a planned economy, of a kind that Marx could only have dreamt of. Because they have records of all transactions, all stocks in the shop”
- Past medical advances promoted global equality by cutting infectious disease and infant mortality in poor regions, benefiting not just recipients but equality worldwide; but future enhancement technologies—genetic modification for intelligence or looks, or radical longevity—if available only to a wealthy subset, would create a new fundamental kind of inequality, reversing the historically equalizing effect of medicine.
“medical advances have been beneficial not just for those who have them, but for promoting equality worldwide, whereas these future ones may have the reverse effect”
- Because we can still appreciate the emotions of ancient Greek and Roman writers—human nature being unchanged—we share affinity across millennia; but human enhancement and machine interfaces could change human nature so drastically that intelligences a few centuries hence may have no more than an algorithmic understanding of us, lacking anything we would recognize as human emotions, which would be a profound game-changer happening over mere centuries rather than evolutionary timescales.
“when we read the literature left by the Greeks and Romans, then we can appreciate it because human nature was the same then”
- Even granting a tiny probability that future technology could revive the frozen dead, cryonic preservation is arguably selfish because, if it worked, the revived would awaken as refugees from the past into an inconceivably different world, imposing a burden on future people who might reasonably feel less obligation toward those who voluntarily made themselves such refugees—possibly just putting them in a zoo.
“they'd be revived into a inconceivably different world. They'll be refugees from the past and they'll be a burden”
- Whether superintelligent AI is a risk or an opportunity is unknown, but the worry that an AI might do things contrary to human interests is not in principle different from the misuse of any other technology, so Rees does not worry much about it as a special category.
“I don't see it in principle different from volume the misuse of any other technology really. So I don't worry so much about that.”
- A major impediment to AI understanding human behavior is a mismatch of time scales: computers learn from millions of examples, but observing human beings is, from the computer's perspective, as slow as humans watching trees grow, so machines cannot accumulate enough data to genuinely model human behavior—making human-general intelligence a far more distant goal than data-heavy tasks.
“watching human beings is like us watching trees grow, it's very slow and they can't accumulate the amount of data in order to really understand that”
Interconnected global systems make societies far more fragile than historical civilizations, with regional disasters now cascading worldwide.
- Modern society is far more fragile than past civilizations because telecommunications, supply chains, and air travel mean any severe regional setback now cascades globally, unlike historical collapses such as those Jared Diamond documented which remained local—and our dependence on systems like electricity means events like a pandemic overwhelming hospitals or a major cyber attack could trigger breakdown of social order within days.
“any really severe setback in any part of the world will have consequences, because we are so interconnected by telecommunications, supply chains, air travel, and everything else, so things would spread globally”
- Sudden catastrophes resemble stock-market crashes in that there is no symmetry between the speed of rises and the speed of falls—we grow complacent during long stable rises and are blindsided by sudden collapses—reflecting the Pareto/fat-tailed nature of such phenomena where outcomes are below average most of the time but the long tail dominates the consequences.
“There's no symmetry between the speed of rises and the speed of falls”
- The consequences of a natural pandemic could be worse than engineered ones: SARS had limited global effect because it spread only to places that could cope, but had something similar reached a megacity of the developing world like Mumbai or Casablanca it could have been very serious—and planning for such events involves non-obvious sociological choices, such as whether to keep mobile phones on to spread information or off to prevent panic.
“SARS wasn't on global effect because it only spread to places that could cope with it... had something that spread to one of the mega cities of the developing world, like Mumbai, or Casablanca... Then it could have been very, very serious”
- The stock market keeps on rising and you think it'll go on forever.
“It's like in the stock market where it keeps on rising.”
We cannot meaningfully quantify probabilities of unprecedented events, so underestimation of novel risks is structural and nearly unavoidable.
- While some events like coin tosses or village-level frequencies permit meaningful probabilistic estimates, the probability of an event that has never yet happened cannot truly be quantified, and this inability is precisely why we tend to underestimate such risks—the probability that our underlying theory is wrong is itself not small, so genuine surprises cannot be ruled out or quantified.
“we can't quantify any event's probability if it hasn't yet happened and that's the reason why we may tend to underestimate”
- It is not stupid to query whether the Large Hadron Collider could create a mini black hole or strangelets that destroy the Earth, because even though the probability is very low, the consequences are catastrophic—and the main risk is not that current theory predicts disaster but that our theory could be completely wrong when exploring a region of parameter space never explored by nature itself.
“obviously it's very very unlikely, but the consequence are catastrophic, that I think it's not stupid to query this”
- During the Cold War the realistic risk of a catastrophic nuclear exchange devastating the northern hemisphere was substantial—on the order of one in three for those in Europe, with figures like McNamara and Kennedy later estimating between one in three and even odds—and had people genuinely appreciated this magnitude, many would have rejected the conventional policy, preferring even a Soviet takeover of Europe to such destruction.
“McNamara in his later years, and Kennedy saying that the risk was between one in three and evens”
Long-term future decisions should weight distant generations heavily, especially where damage is irreversible, unlike commercial discounting.
- It is only apparently paradoxical that medieval people with limited horizons built cathedrals they would never see finished while we hesitate to plan so far ahead: medieval builders, though they thought the world might end, were confident their grandchildren's lives would resemble their own and would appreciate the cathedral, whereas we have little confidence predicting everyday life even 50 years out given recent rates of change—so it is rational to plan less far ahead in some contexts, except where irreversible damage is at stake.
“in medieval times, even though they thought the whole world might end, they thought the lives of their children and their grandchildren would be more or less like theirs. So the grandchildren would appreciate the completed cathedral”
- Decisions affecting the long-term future should not discount future generations as heavily as a commercial transaction does, because while a commercial building decision requires payback within 30-40 years, people genuinely care about the life experiences of babies born today who will live into the 22nd century, and for irreversible long-term damage we surely ought to weight future generations heavily.
“there is a question of what discount rate you apply and how much we should think about future generations rather than those who are now alive”
- Effective action on long-term global threats like climate change is hard to obtain because politicians naturally focus on the immediate and parochial—thinking only up to the next election and about their own constituents—which makes prioritizing collective long-term global action structurally difficult.
“politicians tend to focus naturally on the immediate and the parochial. They think up to the next election and they think about their own constituents”
Scientists must make reasoning on catastrophic risk publicly transparent and be independently vetted rather than merely self-assuring.
- When scientists who have a self-interest in performing an experiment claim it is safe, there is an obligation to have an independent group without that self-interest reassure the public, and more broadly scientists must make their reasoning about catastrophic risk publicly transparent rather than asking the public to simply trust the scientists.
“it's good to have some independent group of people who can sort of reassure themselves without having any self-interest in the experiment”
- Scientists can have more effect on long-term policy by going public and becoming 'small-time Carl Sagans' than by serving as in-house government advisors, because politicians respond to what voters want; if the public is made to care about an issue, politicians will feel safe acting on it without losing votes—exemplified by the Pope's 2015 climate encyclical easing the path to the Paris consensus and the David Attenborough Blue Planet II albatross imagery driving UK plastics legislation.
“scientists can have more effect if they go public and become small time Carl Sagan's as it were because then they do have an influence. And politicians will respond if they think the voters are behind them”
Nuclear power should be part of clean energy, requiring modern R&D in fourth-generation and small modular reactors to compete effectively.
- Although Britain produces only about 1% of world CO2 emissions—so cutting them to zero is negligible globally—Britain has historically produced over 10% of the world's clever ideas, so a crash R&D program in clean energy storage that lets developing nations like India leapfrog directly to clean energy would reduce world emissions by far more than 1%, making R&D leverage the most effective contribution a small advanced nation can make.
“we're only 1% of the world's emissions, so it's neither here nor there but I think we in Britain can claim to have produced more than 10% of the world's clever ideas up till now”
- Nuclear power should probably be part of the clean energy answer, but the field has suffered from no real R&D in 20 years with designs dating to the 1960s; developing fourth-generation nuclear and small modular reactors—small enough to transport on a truck and potentially mass-produced—would be safer and more economical, and concerns about nuclear waste are solvable while people worry too much about low radiation levels.
“the problem of nuclear is there's been no real R&D in the last 20 years. The designs being used really date back from the 1960s”
Global regulation of bio and cyber technology faces fundamental enforcement gaps that simple rules cannot overcome across diverse jurisdictions.
- Bio and cyber regulation creates a growing three-way tension between liberty, privacy, and security, and because different parts of the world will balance these differently—the Chinese caring less about privacy and more about security than the US, with Europe in between—global regulation of these domains is fundamentally difficult.
“there is going to be a growing tension between three things we want to preserve; liberty, privacy and security”
- One of the most depressing features of technological development is that the fraction of effort devoted to security—everything that would be unnecessary if everyone were honest—is getting larger and larger.
“the fraction of efforts that go into security and all the things that wouldn't be needed if we were all honest is getting larger and larger”
Extraterrestrial intelligence, if detected, would likely be electronic artifacts rather than biological civilizations due to developmental desynchrony.
- Finding evidence that life originated twice independently within our own solar system would immediately tell us that life of some kind is widespread across billions of places in our galaxy; this is why searching for even primitive life is so important, but the second origin must be convincingly independent—Mars is less convincing because life could have transferred via meteorites ('we may all be Martians'), whereas life on Europa, or with very different chemistry, would establish genuine independence.
“if we could find evidence that earth... Life had originated twice independently within our solar system, that would tell us straight away that life of some kind was widespread in billion places in our galaxy”
- If we ever detect extraterrestrial intelligence, it will almost certainly not be a flesh-and-blood civilization on a planet but some electronic artifact, because any planet undergoing Earth-like evolution would likely transition within a few centuries to electronic entities with billion-year futures; two planets' developments are extremely unlikely to be synchronized to within the few millennia of a technological civilization, so we would catch another world either far behind (just a biosphere) or far ahead (electronic artifacts).
“it won't be any sort of flesh and blood civilization on a planet, it will be some sort of maybe burping and malfunctioning artifact probably roaming in space”
- Searches for extraterrestrial intelligence—radio transmissions, optical flashes, or artifacts—deserve modest funding and effort despite very long odds, both because the question fascinates everyone and because the potential significance is so great, even though a big radio telescope listening for signals is a very long shot since an advanced civilization would have little reason to beam radio in random directions.
“I'm an enthusiast for modest expenditures on all these possible searches because it's a fascinating to all of us”
- The Fermi Paradox is not watertight because Darwinian selection favors both intelligence and aggression/expansionism, but electronic intelligences that did not evolve by Darwinian selection may be entirely contemplative and non-expansionist; thus the galaxy could contain advanced electronic intelligences that do not manifest their presence conspicuously, even though it cannot contain many expansionist civilizations since one would already have reached us.
“we have evolved by Darwinian selection, which favors intelligence but also favors aggression”
- The origin of life, long relegated to the 'too difficult box'—evidenced by the 1950s Miller-Urey experiment still being the reference point 40 years later—is now drawing serious top-ranking biologists motivated by microbiology advances and the realization of other galactic habitats, making a plausible scenario for life's origin likely within 10-20 years, which would reveal both whether it was a rare fluke and whether it depends on specific chemistry like DNA/RNA or even on water as a solvent.
“the Miller-Urey experiment done near 1950s... was still talked about 40 years later. No one thought of doing much better experiments”
- Within 10-20 years, instruments like the James Webb Telescope and especially the Extremely Large Telescope—a 39-meter mosaic of 800 glass pieces—should collect enough light to spectroscopically separate a planet's reflected light from its star, revealing biospheres via signatures like chlorophyll; and looking further ahead, robotic fabrication of a several-hundred-meter lightweight mirror in zero gravity could even resolve an exoplanet as an extended image rather than a point.
“this has a 39-meter mirror which is not one sheet of glass but a mosaic of 800 pieces of glass, and this will collect enough light such that with the right spectograph, it should be able to separate out the light reflected from the planet from the billions time brighter light from the star”
- Some subjects are so far from any kind of empirical test that they are not worth discussing and should be left in the 'too difficult box'—this was once true of questions about extraterrestrial life, but exobiology is now a serious empirical subject rather than a flaky one, illustrating that the boundary of legitimate inquiry shifts as empirical access improves.
“there are some subjects which are so far from any kind of empirical test, that it's not worth discussing them, and that should be left in the too difficult box and left alone”
High-energy physics risks are allayed by nature already running equivalent experiments through cosmic rays without catastrophic outcomes.
- Many fears about novel high-energy experiments can be allayed because nature has effectively already run the experiment: high-energy cosmic rays have collided without disaster, and the continued existence of stars and white dwarfs that have not turned into strange matter provides empirical reassurance—though genuine novelty (effects never occurring in nature) warrants open-minded caution.
“Cosmic rays of high energy have collided and nothing disastrous has happened, and the fact that we can see stars, white dwarfs that are not turned into strange that's tell us something”
Crewed spaceflight lacks practical justification when robotics suffice; space exploration should be left to private adventurers accepting risk.
- Mass emigration to space as a survival strategy is a dangerous delusion because dealing with climate change is trivial compared to terraforming Mars; there is no Planet B for ordinary risk-averse people, since Mars would be less comfortable than the South Pole—we should encourage adventurers to go, but not pretend space offers a refuge for humanity at large.
“dealing with climate change, for instance, is a challenge, but it's a doddle compared to terraforming Mars”
- With every advance in miniaturization and robotics, the practical need for humans in space diminishes, so there is no practical case for taxpayer money on NASA's crewed program; the future of space science lies in swarms of miniaturized robotic probes—and crewed exploration should be left to private adventurers prepared to accept the high risk that made NASA's program so expensively risk-averse.
“with every advance in miniaturization of robotics, the practical need for the people gets less”
- The special perspective an astronomer brings to humanity's future is awareness of the vast ranges of future time and extended space: it took 4 billion years for us to emerge but the Earth has 6 billion years ahead and the universe far longer, so no astronomer could believe humans are the top of the tree or the culmination of evolution—we are not even at a halfway stage in emergent complexity.
“It's taken 4 billion years for us to emerge from the primordial slime, as it were, but the time ahead, even for the Earth, is 6 billion years”