explainability
The ability to understand and articulate why an AI system produces particular outputs, by converting opaque internals into human-comprehensible form.
As defined in Roman Yampolskiy
Other definitions
The ability to provide clear, understandable explanations for why a system (particularly an AI system) made a particular decision or took a specific action
The Real Risk of Artificial Intelligence | Matthieu Ricard
Where it comes up · 3
- Roman Yampolskiy3 claims
- Future of Finance1 claim
- 148 - Jack Dorsey1 claim
Most important claims · 5 of 5Search all claims for “explainability” →
- One of the most dangerous aspects of advancing AI is that many algorithms cannot explain why they made decisions or what criteria they used, so as both Twitter and society export more decisions to algorithms, the field of explainability—being able to state the 'why' behind every action—becomes critical.148 - Jack Dorsey
- In finance no one can rely on a machine recommendation without a trail of reasoning, so encoding human knowledge is needed to explain in plain English why a machine said to buy or sell, which pure machine learning usually cannot do.Future of Finance
- Sufficiently large models cannot be completely explained because no single human has the capacity to comprehend what trillions of connections represent; the only true explanation is the model itself, and any compressed explanation is incomplete, while targeted deception can tailor explanations so different individuals are deceived in different ways.Roman Yampolskiy
- Rather than a time-based pause, AI development should be paused until specific safety capabilities (explainability, prediction of all intermediate steps, control, unambiguous communication, verification) can be achieved—if achievable, development proceeds, but if they are impossible as Yampolskiy believes, it becomes a permanent ban.Roman Yampolskiy
- It is very hard to separate capabilities work from safety work, because progress in explainability—converting opaque neural network weights into manipulable, human-understandable code—makes self-improvement and intelligent design (rather than gradual gradient descent) much easier, thereby greatly increasing capability and danger.Roman Yampolskiy