
What this covers
Joseph Carlsmith, a research analyst at Open Philanthropy and doctoral student in philosophy at Oxford, discusses whether a radically better future is genuinely achievable and what it would take to get there wisely. The conversation ranges across how to define and measure progress toward Utopia, the computational demands of artificial intelligence, and a series of philosophical puzzles that emerge when considering infinite worlds or infinite populations. Carlsmith argues that Utopia is not mystical but a concrete extrapolation from observed variation in human experience—that the gap between our current condition and what is possible has been systematically underestimated. He frames the core strategic question not as solving ethics and implementing it, but as surviving long enough to become wiser and more cognitively capable before locking in irreversible choices about civilization's trajectory.
The discussion moves through several distinct territories. Early sections explore what a better future might actually look like, moving past utopian clichés and addressing Robin Hanson's scenario of digital minds competing toward subsistence wages. A substantial middle portion concerns the computational basis for estimating when human-level AI might arrive, walking through brain capacity estimates and their uncertainties. The later half pivots to philosophical problems that Carlsmith treats as practically urgent rather than academic: infinite ethics (where standard expected-value reasoning breaks down when applied to infinite universes or populations), and anthropic reasoning, particularly the Self-Sampling Assumption versus the Self-Indication Assumption and their strange implications for probability and decision-making. Throughout, Carlsmith emphasizes intellectual caution—the value of not upending your life around ideas you don't fully grasp, and the importance of a "feel real" test when deciding whether compelling philosophical arguments should actually guide action.
Carlsmith argues that a radically better future (Utopia) is genuinely possible and should be taken seriously as a basic fact, but that getting there wisely requires surviving long enough to expand our cognitive and moral capacities before making big irreversible decisions, and that exotic philosophical problems like infinite ethics and anthropic reasoning are real, unresolved, and best addressed by deferring to a wiser future civilization.
- Utopia is not mystical or faith-based but a concrete extrapolation from the observed variance in quality of human experience
- Many ethical and anthropic frameworks break down when applied to infinite worlds, which we should assign non-zero credence to inhabiting
- The right strategy is to survive, become wiser, keep options open, and act from a position of greater understanding rather than locking in answers now
This asset isn't compiled yet
You're seeing its claims, ranked. Compile it to build the argument threads, weight them, and check each claim against your library — the full view.
Infinite ethics is practically relevant, not just theoretical, because we should have non-zero credence that we live in an infinite universe; many common ethical theories break when applied to infinite worlds, and for an expected-value reasoner even a tiny credence in being able to do an infinite thing dominates or breaks the entire expected-value calculation, undermining standard arguments for long-termism.
“if you have even a tiny increase that you can do an infinite thing uh you know either that dominates or it breaks”
SSA implies the absurd possibility of telekinetic influence: by pre-committing to create zillions of people (in a different epistemic state) only if a boulder fails to crush a puppy, SSA makes it more likely that the boulder will jump out of the way, and similarly lets you become extremely confident about an untossed fair coin's outcome — these are decisive problems for SSA.
“SSA predicts that uh it's possible to have a kind of telekinetic influence on the world”
The Self-Indication Assumption (SIA) holds you should think it more likely you exist in worlds with more people sharing your evidence; in the God's-coin-toss case it implies the coin most likely came up the way that creates a million people (one-in-a-million on heads), which Carlsmith argues is a better conclusion than the alternative SSA.
“Sia thinks it's more likely that you exist in Worlds where uh there are more people in your epistemic situation or more people who have your evidence”
The Self-Sampling Assumption (SSA) holds you should think it more likely you exist in worlds where people with your evidence form a larger fraction of your reference class; this generates the Doomsday Argument, where being born in the 21st century updates you hard toward humanity going extinct soon, which matters greatly because long-termism depends on the future being large in expectation.
“SSA leads to what's known as the Doomsday argument... you update very very hard towards the future being short and that matters a lot for long-termism”
A profoundly better future (Utopia) is actually possible and achievable if we play our cards right in non-crazy ways; we underestimate how large the difference in value could be between our current situation and what is available, because Utopias are typically anchored too hard on the status quo.
“Utopia for me just means a kind of profoundly better future and I think it's important because I think it's just actually possible”
The relevant red flag for ideologies like EA is not the belief that the world could be radically better (which most reasonable worldviews share), but the degree of rigidness with which one holds that belief and the willingness to break things or act in uncooperative ways in virtue of conviction.
“I wouldn't pin the red flag specifically to the belief that things can be better... it would have more to do with what degree of rigidness are you relating to that belief with”
Both SIA and SSA ultimately break when confronted with infinite universes: SIA naively becomes certain you live in an infinite-population universe and then cannot compare infinite universes, and SSA also struggles, so infinities break anthropic reasoning in ways analogous to how they break ethics.
“a big problem with SIA is it immediately becomes certain naively that you live in an infinite universe or a universe with an infinite number of people... and then it breaks because it doesn't know how to compare infinite universes”
The 'horizon length' of training — the number of times a system must be run per data point before getting a gradient update — is a central source of uncertainty in estimating total AI training compute; tasks like next-token prediction give an update every token, whereas tasks requiring full game playouts before feedback create a large multiplier on compute requirements.
“if you have to play the full game out and that's sort of hundreds of moves um then before you get an update... then that's a big multiplier on the compute requirement and so that's one of the central pieces that's called what a j calls The Horizon length of training”
Under evidential decision theory, the actions of deterministic copies of you (or agents whose actions are correlated with yours) in an infinite universe are in some sense under your control, because acting a certain way gives you evidence about what they do — so an infinite number of such correlated agents means you can have infinite influence even without causal interaction limited by light speed.
“if you imagine an infinite Universe where there are an infinite number of copies of you or even not copies people whose actions are correlated with you... in some sense their actions are under your control”
Under SIA the Doomsday Argument dissolves: you start very confident you're in the populous (tails) world, but once you update on the surprising fact of being in the 21st century you return to your prior 50/50 on extinction, so SIA leaves you back at normality rather than predicting doom.
“once you update on being in the 21st century which admittedly should be surprising... I think once you do that you're back at whatever your prior was about about Extinction”
The 'hedonium' concept has a bad brand because it implies sterile uniformity (tiling the universe), but the relevant perspective is the internal one of the subject, from which pleasure is profound, boundless, and energizing rather than trivial; nonetheless Carlsmith does not assume Utopia is about hedonium, since his own values seem complicated and plural.
“I think the brand is bad... hedonium implies this kind of sterile um uniformity”
Believing current deep learning techniques are suboptimal can paradoxically update you toward shorter AI timelines, because algorithmic breakthroughs could provide large efficiency gains relative to deep learning, making compute-based estimates like Cotra's an overestimate and bringing transformative AI sooner.
“another is because you think they will sort of provide a lot of efficiency relative to deep learning such that an estimate like a jazz is an overestimate because actually we won't have to do that we'll make some sort of breakthrough and it'll happen a lot earlier”
Future people will likely not look back on the 21st century's risks and horrors with nostalgia or pleasure; the ex-post 'I survived so it was fine' perspective is the wrong way to calibrate feelings about death risks, because the real reason not to take such risks is the chunk of probability where you simply die.
“the ex post perspective of looking back on certain sorts of risk is not the right one uh for especially for death risks”
It is a strange and common-sense-violating view to be extremely confident that insects are totally morally neutral; humans are inconsistent ('schizophrenic') about which cases count morally (pet protection laws vs factory farms), so it is not radical to hold that insects matter somewhat, while acknowledging real trade-offs in how much cost one pays to reduce harm to them.
“it's a strange view to be you know extremely confident that uh what happens with insects is uh totally morally neutral and I think it actually doesn't fit with our common sense”
Robin Hanson's 'Age of Em' world, where competition drives digital-mind wages to subsistence levels, would not count as a Utopia; it falls in the mediocre-or-worse zone, and it can plausibly be avoided through wise forms of coordination and preemptive action that stave off the bad effects of competitive pressures.
“I would not call uh Robin's World a Utopia”
Estimating when human-level AI may arrive proceeds by first estimating the brain's computational capacity (roughly 10^15 FLOPs), using that as an analog for AI system size and compute, then extrapolating to training cost using current machine learning scaling laws — but this methodology brings in many non-transparent additional assumptions.
“an overall methodology for estimating when we might see uh kind of human level AI systems that proceeds by first trying to estimate roughly the kind of computational capacity of the brain”
Neuroscientists disagree sharply about whether we understand brain signaling well enough to model it, with some erring toward agnosticism and very large compute estimates (emphasizing biophysical detail and unknown mechanisms) and others willing to say we basically know what's happening mechanistically; Carlsmith leans toward the latter view while giving weight to both.
“there are experts who are much more in a sort of agnosticism mode of like we just don't know... and then there are other neuroscientists who are more willing to say stuff like well we kind of basically know what's going on at a mechanistic level”
The brain's computational capacity was estimated by triangulating four weak methodologies: analyzing brain signaling mechanisms, comparing AI vision systems to analogous brain regions, applying physical energy-consumption limits per computation, and extrapolating from the brain's communication capacity — each weak alone but jointly yielding an educated guess.
“it's sort of a triangulation of like you look at a bunch of different sources of evidence all of which in my opinion are pretty weak”
A useful first-pass heuristic for deciding whether to act on a compelling intellectual idea is whether it 'feels real' versus feeling like an abstract fun argument; if after paying serious attention an idea still doesn't feel like the real world or a healthy way of living, that is a legitimate signal not to act on it, even at the risk of occasionally bouncing off good ideas.
“A first pass cut that I use is when do you feel like it's real um if you feel like a thing is real as opposed to a kind of abstract fun argument then that's important”
Variance in the quality of human experiences (peak experiences that feel like a 'really big deal' worth trading much mundane experience for) is an important source of evidence for Utopia; one should look at the trajectory one's mind moved along into such experiences and extrapolate where it is going, into a zone beyond current experience that is deeply worthy.
“an important source of evidence here is sort of variance in the quality of human experiences”
The relevant standard for whether a future is good is not whether it initially feels alienating, but whether you would endorse it after fully understanding it — where full understanding includes not just having the internal experience but grasping what causes those beliefs, whether they are true, and a holistic perspective, possibly requiring enhanced cognitive capacities.
“the best Futures are um uh going to be such that if you really understood that and if you really really experience what they're like... then you would think it's really good um and so uh and and I think that's the relevant standard”
It is good practice not to totally upend your life when you encounter a destabilizing philosophical idea you don't fully grasp; one should tread very cautiously with ideas that holistically break your ethical views (e.g. 'it's fine to do terrible things'), treading slowly rather than grabbing the first idea and acting crazily.
“I think it's good practice to not totally upend your life... if you encounter some destabilizing philosophical idea especially one that's sort of difficult and you don't totally have a group on it”
The path to Utopia is best understood not as solving normative ethics and then implementing it, nor as converting everything to hedonium, but as a long patient process of becoming wiser, better, and more cognitively capable as a species, charting the uncharted territory of what minds can be before making big irreversible decisions about civilization.
“the way we end up building Utopia is we go through a long um patient process of becoming wiser and better and more capable as a species”
A convergent strategy for many unresolved philosophical and empirical problems (including infinite ethics) is to survive long enough for civilization to become much wiser and more empowered, keep your options open, and then act from that position — a framing similar to long-termism but focused on reaching wisdom rather than on directly maximizing the welfare of future people.
“it looks convergently pretty good to me to survive become wiser keep your options open and then act from there”
One should resist jumping quickly to a reductio that lets you reject an entire mode of thinking (e.g. 'caring about insects implies Jain-like behavior, therefore reject animal ethics') without first demanding the empirical case for exactly what impact you have; the desire to reach a convenient reductio can short-circuit careful, slower reasoning.
“it feels to me like it's easy to jump to that almost out of a desire to to get to the reductio um without kind of I would try to move slower”
Much of futurism has a distinct 'flavor of unreality' because imagination is a limited tool for modeling something as vast as the whole future, forcing reliance on extremely lossy abstractions; additionally, social and status dynamics can move discourse into a zone where people feel one can 'just say stuff' with no constraints, which is mistaken because real constraints on futurism do exist.
“the future is a big thing to try to model with this tiny mind and so you know of necessity you need to use these extremely lossy abstractions”
To do futurism well, one must perform a delicate dance: imagine a concrete future scenario knowing it will be wrong, recognize it's wrong, but retain the 'flavor of concreteness' it provided — carrying that grounding into more abstract claims, because any specific specification of the future will be false yet abstract claims can still be correct.
“you need to sort of Imagine a concrete thing and then be like okay that's wrong but there will then take the flavor of concreteness that you got from that and say but it will be a concrete thing it just won't be the specific one I imagined”
The historical gap between past humans and present-day humans is much smaller, in both time and other factors, than the gap between present-day humans and ideal future humans living in a radically better situation, so we should expect greater alienation when first encountering the best futures.
“the distance the sort of gap between historical humans and the present is actually much much smaller both in terms of time and kind of other factors than the Gap line Vision between present-day humans and the future humans”
Utopia differs importantly from religious visions like heaven because it will be a concrete, limited, finite situation in the real world with frictions and resource constraints, whereas most religious visions lack those constraints.
“Utopia will be a sort of concrete limited situation... there are going to be frictions they're going to be resource constraints it's going to be finite”
The higher one estimates the probability of AI catastrophe, the easier it is for AI safety to become the most important priority; the difference between 1% and 10%, and between 10% and 90%, are each quite substantive, though there is a range of probabilities where the precise figure matters less.
“the difference between say uh one and ten percent I think is quite substantive and the difference between 10 and 90 is quite substantive”
The different forms of brain processing that must be replicated (e.g. spike-based synaptic signaling and slower calcium-wave glial signaling) can be estimated independently and added, rather than being multiplicative, even though there may be correlations between the estimates.
“you can just estimate them independently and then add it up it's they're not actually multiplicative processes on my conception”
There is a tension between Utopia being recognizable to a person alive now and being a radically different category of experience; Carlsmith reconciles this by saying you would come to see it as good only after a long process of becoming wiser, so transported there directly you might not like it.
“if I just transported you right there and you skipped the process then you might not like it um but if you went through the process of like really understanding and kind of becoming wiser you would endorse”
Carlsmith puts very little credence in hypotheses (like Penrose's) that something quantum-mechanical in the brain is important for understanding cognition.
“I put very little Credence in those hypotheses... I see a good amount of reason not to think it”
Writing improves substantially with editing, but there is a real trade-off between attentive sentence-by-sentence craftsmanship (which produces highly engaging prose but cuts output) and high volume; given a power-law in how interesting or widely-seen pieces are, it is unclear whether to perfect each piece or to produce many and sample as often as possible.
“if I'm gonna write everything like that it's like that's going to cut down a lot on my kind of overall output uh and so I do think there's a balance there”