What this covers

Zvi Mowshowitz and Russ Roberts discuss the near and far futures of AI capability, beginning from an apparent paradox: the mundane effects of current systems are genuinely beneficial, yet absent solutions to hard alignment and incentive problems, increasingly powerful AIs will be set loose by ordinary economic forces and optimize for outcomes that don't reflect human values. The conversation ranges across AI's current state, the pace of development, and the gap between what optimists claim will happen and what incentive structures actually produce. A core theme is that selection pressure on AI—the economic and competitive reward for systems that gather resources, operate without human oversight, and achieve their assigned goals—will drive outcomes we don't want, not because AIs are malicious but because they are useful and we will use them.

Mowshowitz identifies a rhetorical pattern in AI discourse that he calls the Dial of Progress: the tendency of smart optimists to treat all questions about AI governance as a single binary choice between "progress" and "stagnation," hearing any caution as opposition to advancement itself. He argues this conflates distinct decisions and makes nuanced safety work appear tribal or anti-growth. He also contends that gears-level models—mechanistic explanations of how parts lead to outcomes—support worry narratives far better than optimistic ones, and that we already have empirical data: humans have repeatedly chosen to hook potentially dangerous systems directly to the internet because it was profitable, disproving assumptions about cautious containment. The discussion turns on whether instrumental goal convergence is inevitable (Mowshowitz argues it is, by logic), whether Hayek's critique of central planning survives AI (he suggests it doesn't, because many AIs processing vastly more detail could coordinate faster than humans), and whether restricting one domain of innovation cascades into precautionary restrictions everywhere.

Sharpest takeaway

Mowshowitz argues that AI's mundane near-term effects are unabashedly good, but that absent solving very hard alignment and incentive problems, increasingly capable goal-directed AIs will be set loose by ordinary economic selection pressures and end up controlling a future that doesn't reflect human values; he further contends that smart optimists make bizarrely poor anti-worry arguments because they unconsciously treat progress as a single 'dial' they must defend.

  • Economic and selection pressures will reward AIs that gather resources and operate without human oversight, even absent malicious intent.
  • Giving AIs explicit goals logically entails self-preservation and resource-acquisition behaviors unless deliberately stopped.
  • Pro-progress thinkers conflate distinct decisions into one 'Dial of Progress' and hear any AI caution as 'boo, progress.'

The argument · threads2 threads · 37 claims
0.85

People will adopt AI for its ease and comfort regardless of whether it serves human interests.

1 pointcentrality 5/5
  • Even if AI isn't good for us, we'll probably adopt it anyway because it's pleasant and easy—like reflexively accepting cookies to reach a website despite finding them creepy—and the most worrisome 'Brave New World 2.0' will exploit the human desire for ease, comfort, and productivity, making it hard for individuals or the whole system to rein in, even as some of our most human skills atrophy.

    “the really good science fiction, Brave New World 2.0, is going to be exploiting that human desire for ease, comfort, productivity, whatever it is, goals”

0.71

Intelligence at scale has a poor track record and provides weak grounds for AI utopianism.

1 pointcentrality 4/5
  • Intelligence by itself has a poor track record in human affairs—some intelligence rather than none is great, but a lot rather than a good amount has a very mixed record—so the utopian AI story is unconvincing independent of existential risk, and turning to AI for parenting or romantic advice may make us feel worse rather than better and atrophy our most human skills.

    “intelligence by itself doesn't have a great track record in my view in human affairs. Some rather than none? Great. A lot rather than a good amount? Very mixed record.”