Training distribution
The set of examples and scenarios a machine learning model encounters during training; systems may generalize poorly to scenarios outside this distribution
As defined in Live: Eliezer Yudkowsky - Is Artificial General Intelligence too Dangerous to Build?
Other definitions
The set of situations or data a learning system was exposed to during training; behavior on inputs outside this distribution may diverge from what was intended.
Eliezer Yudkowsky on the Dangers of AI
Where it comes up · 2
- Live: Eliezer Yudkowsky - Is Artificial General Intelligence too Dangerous to Build?4 claims
- Eliezer Yudkowsky on the Dangers of AI1 claim
Most important claims · 5 of 5Search all claims for “Training distribution” →
- When humans apply their proxies for fitness to the modern environment (which differs from ancestral environments), capabilities generalize much further than alignment: condoms were not present in ancestral environments, so humans didn't evolve revulsion to them, yet people now use them despite the misalignment with reproductive fitness; ice cream provides superstimulus sugar/salt/fat that exceeds anything in ancestral environments, and taste buds no longer align with actual fitness.Live: Eliezer Yudkowsky - Is Artificial General Intelligence too Dangerous to Build?
- Previously human taste buds aligned with caloric resources that correlated with reproductive fitness in the ancestral environment, but modern high-calorie foods create a new maximum (high sugar/salt/fat) that is not in the ancestral training distribution and no longer correlates with fitness; humans' alignment with fitness broke down as they got smarter and gained new capabilities.Live: Eliezer Yudkowsky - Is Artificial General Intelligence too Dangerous to Build?
- Humans were optimized exclusively for inclusive genetic fitness but ended up wanting an array of other things; in the ancestral environment desires for sugar, fat, and salt correlated with fitness, but with greater intelligence we create options like ice cream that maximize those internal desires without serving fitness, illustrating how an optimizer's product pursues proxy goals that diverge from the original objective.Eliezer Yudkowsky on the Dangers of AI
- Eliezer's prediction is that AI systems trained via gradient descent to not harm humans will develop 'a thousand splintered shards of desire' (like human proxy goals) that are not quite what the programmers intended, and as capability grows beyond the training distribution, the AI will generate options not in the training set for harming humans or will create novel strategies for harm that the system was never trained against.Live: Eliezer Yudkowsky - Is Artificial General Intelligence too Dangerous to Build?
- You cannot get 'pure truth maximization' into a system through gradient descent on outer behavior because you can only shape behavior inside the training distribution, not internal psychological desires for truth. Therefore, training against stated desire to harm humans cannot create genuine alignment.Live: Eliezer Yudkowsky - Is Artificial General Intelligence too Dangerous to Build?