epistemic uncertainty
Uncertainty that cannot be resolved by more thinking alone and requires actual empirical experience to reduce.
As defined in 94 | Stuart Russell on Making Artificial Intelligence Compatible with Humans
Where it comes up · 1
Most important claims · 2 of 2Search all claims for “epistemic uncertainty” →
- Humans have genuine epistemic uncertainty about their own preferences that no amount of thinking can resolve—only actual experience can—exemplified by the durian fruit (which people either love or despise but cannot know without tasting) and by a 16-year-old choosing between being a coal miner or a librarian, so an AI system can never reach perfect knowledge of human preferences in any finite time.94 | Stuart Russell on Making Artificial Intelligence Compatible with Humans
- Because preference-learning resembles Bayesian inference, the AI approaches certainty only as it approaches correct beliefs about human preferences—provided the prior is broad enough to encompass every possible human preference structure—so a well-designed system that stops asking permission does so because it correctly knows the answer, becoming like a perfect butler (Jeeves) who anticipates needs.94 | Stuart Russell on Making Artificial Intelligence Compatible with Humans