interpretability
The degree to which one can understand what a model is actually doing internally to produce its outputs.
As defined in 301 | Tina Eliassi-Rad on Al, Networks, and Epistemic Instability
Other definitions
The ability to understand and explain how an AI system works, why it makes particular decisions, and what mechanisms drive its behavior. Modern AI systems have poor interpretability because they consist of billions of numbers (weights and parameters) whose interactions are opaque.
Why this top AI guru thinks we might be in extinction level trouble | The InnerView
Where it comes up · 19
- The AI Tsunami is Here & Society Isn't Ready | Dario Amodei x Nikhil Kamath | People by WTF4 claims
- Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future3 claims
- Eliezer Yudkowsky — Why AI will kill us, aligning LLMs, nature of intelligence, SciFi, & rationality3 claims
- Dwarkesh Patel Podcast - Paul Christiano – We could be 12 months away from AIs that can overthrow the government3 claims
- Quant in Private Markets2 claims
- Data, Decisions, and Basketball2 claims
- Paul Christiano — Preventing an AI takeover2 claims
- Demis Hassabis — Scaling, superhuman AIs, AlphaZero atop LLMs, AlphaFold2 claims
and 11 more
Most important claims · 10 of 32Search all claims for “interpretability” →
- Modern machine learning models are too complex to read individual variable coefficients, so practitioners must build side models for interpretability and accept that they do not fully understand how the system works.Data, Decisions, and Basketball
- Interpretability work by Chris Olah and others at Anthropic trained a very sparse autoencoder that could deduce what characteristics a particular neuron has—finding, for example, a Golden Gate Bridge neuron activated when discussing the Golden Gate Bridge—and similar approaches could work at the expert level, though such deep interpretability may not be necessary if the model's collective output is good, which is one of the beauties of deep learning.Jeff Dean & Noam Shazeer — 25 years at Google: from PageRank to AGI
- The critical point of failure is when AI hacks the servers it runs on and subverts the software controls (weight updates, interpretability tools, lie detectors, supervision), because all our control mechanisms operate in computers; once that hard power is lost—possibly early and invisibly—everything afterward appears to be going well while the AI sets up the actual takeover.Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future
- We can get rich empirical feedback on AI safety by setting verifiable challenges—e.g. giving an AI control of an air-gapped computer's keyboard and asking it to root the environment to make a blue banana appear—because we can see whether it succeeded even if we'd never have detected the particular exploit, letting us test whether our interpretability and adversarial methods catch an AI using its best efforts.Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future
- Because of hype, engineers often fail to state their assumptions and technical limitations, producing not merely a replicability problem but a reproducibility problem—an inability to even rerun the same code on the same training data with the same folds—which is a very low bar that nonetheless frequently fails.301 | Tina Eliassi-Rad on Al, Networks, and Epistemic Instability
- There is a band of AI capability—the 'AI for AI safety sweet spot'—where AIs are capable enough to massively accelerate security-relevant factors like alignment research, control, cybersecurity, and epistemics, but not yet capable of taking over the world, and if civilization commits hard to redirecting that productivity toward security we have strong tools to stay safe.Joe Carlsmith — Preventing an AI takeover
- Every AI company should publish a public 'AI constitution' or model spec describing how and toward what character and rules they train their models, and—ideally backed by regulation—should be required to demonstrate to a high evidentiary standard that the AIs were in fact trained in accordance with that constitution, including that they aren't backdoored or secretly loyal to particular people; OpenAI and Anthropic publishing such documents is a praiseworthy first step.467 - Ea Ai And The End Of Work
- Because reinforcement has dangerous aspects—rewarding apparently-good behavior can train deception—it is more robust to verify the system's reasoning process and its understanding of ethics directly, grilling it to pull apart its understanding and having people continuously review its decisions and reasoning during deployment.Shane Legg (DeepMind Founder) — 2028 AGI, superhuman alignment, new architectures
- Anthropic has pioneered the science of interpretability—the ability to look inside neural networks and identify neurons and neural circuits that correspond to specific concepts, such as neurons tracking rhyme patterns in poetry, providing a method to understand what models are actually doing rather than treating them as black boxes.The AI Tsunami is Here & Society Isn't Ready | Dario Amodei x Nikhil Kamath | People by WTF
- Anthropic has released a constitution for Claude, a method to align models in line with a predefined constitution, representing a structured approach to ensuring models behave as intended rather than relying on ad-hoc training methods.The AI Tsunami is Here & Society Isn't Ready | Dario Amodei x Nikhil Kamath | People by WTF