factual

Smarter minds shift goals but arbitrary minds don't get nicer

The orthogonality thesis holds that almost any self-consistent utility function can pair with a self-consistent mind, so smart things are not automatically nice; taking a human and making them smarter may shift and even improve their morality because the niceness was already inside them (like pebble-sorters getting better at finding correct heaps), but this betterment-through-enlightenment does not generalize to arbitrary minds whose preference updates differ or who have no such updates at all.

factualpending

Speaker

Eliezer Yudkowsky

Evidence Quote

“the problem is that there’s other ways for minds to hold together coherently, where they execute other updates as they know more or don’t even execute updates at all because their utility function is simpler than that.”

Source

Eliezer Yudkowsky — Why AI will kill us, aligning LLMs, nature of intelligence, SciFi, & rationality— Dwarkesh Patel Podcast
Created: 6/13/2026, 3:29:05 AM

Your notes

Loading notes...