factual

Utility functions are a lossy approximation of agency

Treating real-world human agents as having utility functions is at best a very lossy approximation that becomes increasingly misleading as agent intelligence rises; agency and planning genuinely produce outcomes (a person searches and ends up with a house and dog; civilization gets a man on the moon) without requiring any consistent utility function, and this picture is more real than utility functions.

factualpending

Speaker

Joe Carlsmith

Evidence Quote

“It's very janky...thinking of humans as having utility functions is at best a very lossy approximation.”

Source

Joe Carlsmith — Preventing an AI takeover— Dwarkesh Patel Podcast
Created: 6/13/2026, 3:37:01 AM

Your notes

Loading notes...