270 | Solo: The Coming Transition in How Humanity Lives
What this covers
Carroll explores humanity's approach to a possible technological turning point through the lens of physics and economics. The episode examines how accelerating innovation in AI, synthetic biology, brain-computer interfaces, and climate response might drive a phase transition into a fundamentally different mode of human life. Drawing on Phase Transition as Social Metaphor, Carroll argues that a singularity need not mean collapse or apocalypse—physical singularities herald shifts to new equilibrium states, and the social analog could similarly mark passage into a radically reorganized but stable world. The crucial question, he contends, is not whether this happens but whether the resulting equilibrium extracts human value dystopically or frees people from toil.
The discussion ranges across how technology—particularly large language models and gene editing—reshapes institutions, labor, and trust. Carroll identifies an Efficiency-Happiness Tension: markets increasingly extract maximum value from transactions through dynamic pricing and targeted advertising, leaving participants worse off in experience even as efficiency rises. He addresses why Self-Interest Under Constraints matters more than individual values, how Pace of Technology vs Pace of Adaptation strains democracy, and Beware Naive Extrapolation from current growth rates. The episode also assesses specific frontiers: synthetic biology's potential to solve climate problems and engineer organisms; the epistemological hazard of AI-generated content eroding trust in real evidence; and why restricting gene editing is likely futile given global competitive dynamics. Throughout, Carroll emphasizes Voice vs Scale—individuals shrink in influence as institutions grow larger—and whether humanity can steer toward an Automated Utopia rather than mere extraction.
Carroll argues that the accelerating pace of technological innovation (AI, biotech, brain-computer interfaces, climate response) is plausibly driving humanity toward a singularity-like phase transition into a new equilibrium of living, and that whether that equilibrium is dystopian (efficient extraction of human value) or utopian (freedom from toil) depends on large-scale social constraints rather than individual choices.
- Singularities in physics are harbingers of phase transitions to new equilibria, not necessarily endings, so we should take the social analog seriously.
- Individual values barely shape outcomes; only large-scale constraints redirect self-interested behavior toward better equilibria.
- Technology makes markets and institutions more efficient at extracting value, which need not increase human happiness.
Singularities in growth signal phase transitions to new equilibria, not system collapse.
- Singularities in physically observable quantities are characteristic of phase transitions, and rather than indicating a system ceases to exist, they signal a dramatic change to a different equilibrium configuration; therefore an approaching technological singularity may herald a phase transition into a new mode of human living rather than an apocalypse.
“Singularities in physically observable quantities are characteristic of phase transitions.”
- Geoffrey West argues, drawing on Will Steffen's 'great acceleration' showing many quantities growing fast, that humanity can avoid hitting the singularity by continually innovating faster and shifting from one growth regime to another as it has before, so even an approaching singularity need not be the end of anything—though Carroll is open to it actually being a phase transition.
“Geoffrey's argument is that we can kind of avoid the singularity by continually innovating. That what we need is faster and faster innovation”
- It is easy but almost useless and dangerous to extrapolate from a current rate of growth, because exponential growth never truly continues forever given finite resources, and apparent exponential growth may actually be a logistic/sigmoid curve that flattens, or a faster-than-exponential pole singularity, and noisy finite data cannot reliably distinguish these.
“Crucially important is that exponential growth never truly continues. There's nothing in nature that grows exponentially forever with the possible exception of the universe itself.”
- The concept of the technological singularity originated in an offhand remark by John von Neumann about humanity approaching an essential singularity in technological progress, was popularized later by Vernor Vinge and Ray Kurzweil in the form of an AI-superintelligence-driven runaway, and von Neumann's use of the technically precise phrase 'essential singularity' may indicate growth uncontrollably faster than any power of 1/x.
“it was first mentioned in a kind of offhand remark by John von Neumann talking to Oolong. And mentioning that humanity was approaching an essential singularity in technological progress.”
AI and synthetic biology will accelerate innovation bootstrapping, producing singularity-like acceleration.
- Not only is humanity innovating at a higher rate, but the pace at which innovations are adopted is itself increasing, as evidenced by ChatGPT being adopted faster than any comparable technology in human history, which makes the data point toward a singularity-like acceleration independent of any particular theory of AI.
“when ChatGPT, the large language model from OpenAI became public, it was adopted faster than any other similar technology in human history, and Geoffrey showed data showing that this is a trend.”
- Synthetic biology—designing new organisms and using DNA's flexibility to build microscopic robots and computers—will be a huge deal, enabling DNA robots that fix health problems, organisms that consume excess atmospheric CO2, and dramatically changed food production, and should be near the top of any list of technologies driving a transition in human life.
“If you're talking about a technological singularity coming that is going to change human life, I think that editing our genes and synthesizing new kinds of organisms had better be right there near the top of your list.”
- Whether or not AGI in the traditional sense ever arrives, AI will absolutely accelerate the rate of innovation by helping humans learn and by simulating expensive trial-and-error processes (like drug trials) in fields such as chemistry, biology, synthetic biology, and gene editing, producing the bootstrapping positive-feedback acceleration characteristic of singularity behavior.
“even if that is not the right way of thinking about it, because the word smart is being misused in that context, the AI will absolutely help accelerate the rate of innovation.”
Optimistic future requires equitable systems distributing technology's gains rather than extractive ones.
- The optimistic scenario is that technology produces excess value and hands off boring, unfulfilling tasks to AIs, cyborgs, and robots, freeing humans to live fulfilling lives—not necessarily writing poetry but in whatever heterogeneous everyday ways they choose—supported by measures like a universal basic income, provided we collectively choose the equitable system rather than the extractive one.
“computers and robots do all the boring things, all the things we don't wanna do... whereas we get to enjoy life and create things.”
- Technological change has always eliminated common jobs and trying to artificially preserve old occupations is doomed; what is genuinely difficult and alienating is not change itself but the uncertainty of the transition—not knowing whether your job or industry will exist in ten years—so reaching a new stable equilibrium on the other side of the singularity, with a settled set of jobs or societal support, would itself be better than the current dynamism.
“What's so alienating right now is the uncertainty. Like you don't know if you're gonna have your job 10 years.”
- Empirical data on the four-day work week are surprisingly positive: companies that pilot it do not suffer overall productivity loss because workers are more energized to be productive in those four days, and they tend to stick with it, supporting the optimistic scenario in which technology's productivity gains are spent on less work rather than more extraction.
“I was a little surprised at how positive the data are about the four day work week scheme. Individual companies have tried it and it makes everything better.”
Efficient markets and AI systems create extraction tensions that erode human wellbeing and trust.
- A crucial effect of technological innovation is that it makes it easier for markets to reach the perfect point of efficiency—extracting the maximum a person will pay right up to their walk-away point—which means transactions occur but no one is happy about them, because there is a real tension between this kind of profit-extraction efficiency and human happiness, as seen in surge/dynamic pricing, baggage fees, resort fees, student loans, and targeted advertising.
“one crucially important aspect of the technological innovations and improvements that we are undergoing is that it makes it easier for markets to reach that perfect point of efficiency where things are sold, but nobody is really happy about it”
- If you try to create new knowledge by throwing random numbers at an AI to generate many candidate works, you face Borges's Library of Babel problem: the good work exists among the outputs but is buried among near-identical nonsense variants, so finding the valuable knowledge versus the nonsense will eventually require human judgment.
“at some point you're in Borges's, Library of Babel... the problem there is you can't find the book.”
- Large language models are good at interpolating—going between existing examples—which is provocative in art but yields less interesting results in prose, and they are poor at creative extrapolation because they are built to predict the predictable next item, creating a tension with what makes writing good; this could be partly resolved by brute-force generating many random variants and selecting the most creative one.
“there is kind of this difference between interpolation and extrapolation. AIs are good at seeing everything written and kind of going between them.”
- Current AIs (large language models) have a fundamentally different way of thinking from humans, and attributing general intelligence, human-like motivations, desires, and values to them is hopelessly anthropomorphic—an overzealous application of the intentional stance and a category error—because they are trained to sound human, and our upbringing trains us to associate human-like speech with intelligence, so fears of a super-intelligent AI dictator misunderstand what these systems are.
“The origin of our motivations and desires and values is just completely disconnected from the way that these AI programs work. It is a category error.”
- AI-generated content creates two epistemic problems: it lets anyone manufacture fake evidence (e.g. videos of people doing things they never did), but the more underappreciated harm is that real evidence becomes less trustworthy, since people can dismiss genuine recordings as AI-generated, producing epistemic fracturing into tribes and bubbles where contrary information is not absent but simply ignored.
“the other problem, which I think is underappreciated, is that real evidence becomes less trustworthy.”
Technological change outpaces human psychological and social adaptation, creating alienation and vulnerability.
- The only certainty is that the future will be different, and the fact that the future is dramatically different within a single lifetime is itself a new condition of human existence; throughout most of history life changed only a little from generation to generation, and humans are not equipped, trained, or practiced to think about the very real possibility of a radically different world a century hence.
“The future was always different throughout human history, but only by a little bit from generation to generation.”
- The pace of technological change is much faster than the pace of psychological change or change in our habits of meeting, talking to, and interacting with people, so the currently alienating effects of information technology may simply reflect that we haven't yet adapted, though it's also possible we've opened a can of worms we cannot manage.
“The pace of technological change is much faster than the pace of psychological change or the pace of our changes in habits, in how we meet people, talk to people, interact and so forth.”
- Large-scale social change requires changing the minds of people on other sides to build consensus, but current discourse shows little attempt to persuade—people mostly mock or argue with opponents rather than win them over—which makes it hard to achieve the agreement needed to move to a better social equilibrium, possibly because the technological world incentivizes conflict over persuasion.
“You need to actually change the minds of people on other sides, and that's something that doesn't happen a lot these days.”
- Because technology makes institutions larger and more interconnected, individuals relatively shrink in their ability to affect the world around them, producing a feeling of powerlessness—you can influence a locally-owned shop but not Starbucks—which, combined with efficient wealth extraction and dimming expectations of rising living standards, strains democracy and liberal society and pushes people toward strongmen and conspiracy theories.
“they will respond to that sort of loss of power by seeking a strong man to rescue them, or by taking refuge in conspiracy theories”
Emerging technologies like gene editing, brain interfaces, and AI pose governance challenges from competitive dynamics.
- The hope that energy/fuel consumption would saturate is belied by the fact that humans continually invent new ways to use energy, and computing is the current example: keeping the cloud running reportedly consumes more energy or emits more than the entire transportation industry, with large language models among the worst offenders, so as AI use expands, finding energy to run it will be a serious problem.
“either the energy consumption or the fossil fuel emission from just keeping the cloud going is larger than that of the entire transportation industry.”
- Brain-computer interfaces and human-machine integration are coming and could be a hundred times more influential than smartphones, because humans already offload cognition to external tools and trust other humans' stored knowledge, and direct neural access to information would massively amplify this extended cognition for better and worse, transforming learning and performance across all fields.
“whatever you think the cell phone has done, I think it's easily imaginable that brain computer interfaces are going to be a hundred times more influential than that.”
- Gene editing of human embryos and designer babies are coming and cannot be prevented because there is no world government to enforce restrictions and competitive dynamics between countries guarantee adoption, and because the technology will be cheap enough to do in a garage rather than requiring a multi-million dollar lab, so the responsible response is to think carefully about what we want those changes to be and to stop abuses rather than the practice itself.
“If one country says we're not going to do it, another country is going to do it. And then the first country is going to say, well, wait a minute, they're doing it. We better start doing it also.”
- Real existential risks—nuclear war, biological warfare, pandemics—exist, but Carroll deliberately sets them aside to focus instead on whether technological change can settle into a quasi-static new equilibrium of living that might be better or worse.
“I'm not going to really worry about existential risks. And those are real. Nuclear war, biological warfare, pandemics.”
Technology's positive effects require human judgment and careful institutional choices.
- Despite his initial reluctance to accept what sounded alarmist and Luddite, the data now support the conclusion that cell phones have had negative effects on young people, making them on average less happy than before—not a necessary connection but a fixable feature of a moment of change in which we haven't yet learned to use the technology well.
“Cell phones have made young people on average less happy than they used to be. And that not a necessary connection, obviously. This is a fixable thing.”
- The personal values of individuals by themselves do not much drive the overall shape of society; values matter only when implemented as large-scale social constraints (laws, tax policy, institutions, or broad social consensus), because given the large-scale constraints individuals will largely pursue their self-interest.
“Encouraging individual people to behave in a certain way is not really going to drive the overall shape of society.”
- The amount of innovation scales super-linearly with population density, so cities produce not just more innovation because there are more people, but more innovation per person, presumably because denser human interaction allows more idea-sharing.
“the amount of innovation scales super linearly with population density. So in other words, not only do you get more innovation in cities because there are more people, but there's more innovation per person”
- The Matrix's depiction of human bodies as batteries is nonsensical thermodynamically, but reinterpreted metaphorically it captures the pessimistic scenario: human capacities become fuel for an unpleasant society in which technology (AI, gene editing, brain-computer interfaces) squeezes individuals for all the system can extract—not because of evil overlords, but because self-interested individuals embedded in the system's incentive structures produce that configuration.
“the pessimistic scenario that I wanna sketch out is one where human capacities, for the most part, mostly become fuel for a rather unpleasant kind of society”
Climate change poses severe suffering and costs but not existential extinction risk.
- The fear that cutting greenhouse gas emissions requires slowing economic growth is contradicted by evidence: in many countries the rate of economic growth has become decoupled from greenhouse gas emissions, with nations lowering CO2 and other emissions while still growing economically.
“in many countries around the world, the rate of economic growth has become decoupled from the rate of greenhouse gas emissions.”
- Climate change will not cause human extinction and so is not an existential risk, but it will cause enormous human suffering, massive economic costs and loss of human wealth, the extinction of other species, dramatic changes to the biosphere, and the loss of much currently-livable land, with poor people hurt disproportionately though even the rich will suffer.
“I don't think that it's going to lead to the extinction of the human race, but it will absolutely lead to the extinction of other species.”
- If the climate models developed over recent decades are wrong, they are wrong because reality is turning out worse than predicted, as evidenced by 2024 global temperature indicators coming in higher than expected.
“if the climate models that we've been trying to work on for the last several decades are wrong, it's wrong because the reality is worse. Than what the model has predicted.”
Embodied cognition makes consciousness uploading impossible; income inequality reflects two economic classes.
- People are not necessarily happier in agricultural societies than in hunter-gatherer ones: hunter-gatherers arguably had more free time, while agricultural societies had more inequality but possibly a higher average standard of living and more reliable food supply.
“Arguably, you have more free time in a hunter-gatherer society. Almost certainly, there is more inequality once you go to the agricultural model”
- While average life expectancy has risen because fewer people die young and we live closer to the upper limit, the maximum human lifespan (around 120 years) has not changed much regardless of society, because improvements in diet, exercise, and medicine haven't extended the biological envelope—so we are not currently on a trajectory toward a major transition in longevity within the next 100 years.
“the maximum lifespan of human beings hasn't actually changed that much. Whether, it's like you think about it as 120 years or something like that.”
- Because human cognition is inextricably embodied—shaped by hunger, thirst, fatigue, homeostasis, and physiological feelings—uploading the information in a brain into a computer chip, while possible in principle and capable of producing a conscious creature, would sever its connection to a body and therefore would not be a person anymore but something profoundly different, so consciousness uploading is not the major coming transition.
“it's not that we can't upload the information into a computer, it's just that it wouldn't be a person anymore. It might be something, but it would be different.”
- The collapse of journalism and newspapers can be substantially attributed to Craigslist, because many newspapers earned most of their revenue from classified ad sections, and once classifieds moved online—better and freely available to everyone, in line with people pursuing self-interest—the newspaper revenue model went away.
“if you want to point a finger at one event that led to the collapse of journalism it was Craigslist. Craigslist the online classified service because many many newspapers actually got most of their revenue from their classified sections”
- Income distribution empirically follows a Maxwell-Boltzmann distribution for the roughly 97% of earners below about three times the median income, then crosses over crisply to a power-law (fat-tailed) decay for high earners, because there are two classes of earners: additive earners who sell consumable goods or time one at a time (analogous to gas molecules exchanging energy), and multiplicative earners (authors, entertainers, capitalists, investors) whose services or capital can be sold or compounded indefinitely.
“It's at about three times the median income level below three times median income it's Maxwell Boltzmann above three times the median income it is a power law decay.”
Sustainable urbanism and decoupled growth show humanity can innovate toward better equilibria.
- Population growth has not been at a constant rate; it grew slowly between the birth of agriculture and the scientific revolution but has been growing much faster since, which is a sign that something more than a constant rate of growth is going on.
“Between the birth of agriculture and the scientific revolution, it grew, but it's been growing much faster. Since the scientific revolution.”
- Per person, living in cities is far more energy efficient than scattering humanity across suburban and rural areas, because urban dwellers use less land, less heating fuel (shared buildings), and drive shorter distances, so the ongoing global urbanization could form a future equilibrium where humanity lives sustainably on the land.
“per person, it is way more energy efficient to have people live in cities. We don't use as much land. We don't use as much fuel to heat your houses because you're living in group buildings”