sparse autoencoder
An interpretability tool that projects a model's activations into a higher-dimensional space to disentangle superimposed concepts into cleanly separated features.
As defined in Is RL + LLMs enough for AGI? — Sholto Douglas & Trenton Bricken
Where it comes up · 4
- Is RL + LLMs enough for AGI? — Sholto Douglas & Trenton Bricken3 claims
- Dario Amodei2 claims
- Jeff Dean & Noam Shazeer — 25 years at Google: from PageRank to AGI1 claim
- Sholto Douglas & Trenton Bricken — How LLMs actually think1 claim
Most important claims · 7 of 7Search all claims for “sparse autoencoder” →
- Dictionary learning is done after training by feeding the model many inputs, taking activations, and projecting them into a higher-dimensional space and back, with a sparsity constraint—it is unsupervised in that you are not telling the model what features to find, only asking it to reconstruct sparsely, and you label the features post hoc, which is its advantage over linear probes that require knowing what you're looking for.Sholto Douglas & Trenton Bricken — How LLMs actually think
- Interpretability work by Chris Olah and others at Anthropic trained a very sparse autoencoder that could deduce what characteristics a particular neuron has—finding, for example, a Golden Gate Bridge neuron activated when discussing the Golden Gate Bridge—and similar approaches could work at the expert level, though such deep interpretability may not be necessary if the model's collective output is good, which is one of the beauties of deep learning.Jeff Dean & Noam Shazeer — 25 years at Google: from PageRank to AGI
- Dictionary learning via sparse autoencoders extracts interpretable monosemantic features from polysemantic networks where none were apparent before — falling out cleanly without specifying categories in advance — which serves as non-trivial validation of the linear representation and superposition hypotheses and scales to production models like Claude 3 Sonnet.Dario Amodei
- Mechanistic interpretability is reverse-engineering neural networks to find their core units of computation; the field exists because neural networks are grown, not built—contrary to the assumption that we understand AI perfectly because we made it—and since Chris Olah co-founded Anthropic three and a half years ago there has been a major breakthrough roughly every six months: toy models of superposition, sparse autoencoders (Towards Monosemanticity), scaling to 30 million features on Claude 3 Sonnet, and now circuits.Is RL + LLMs enough for AGI? — Sholto Douglas & Trenton Bricken
- Counterintuitively, larger models share more representation across languages and modalities in an abstract space rather than separating concepts into distinct neurons; despite having more capacity to separate things, bigger models pull on better, shared abstractions—e.g. the same neural pattern represents the Golden Gate Bridge as text or image, and notions like large/small or hot/cold are shared across languages.Is RL + LLMs enough for AGI? — Sholto Douglas & Trenton Bricken
- Contrary to the meme that neural networks are over-parametrized, for hard tasks like predicting the next token of the entire internet models don't have enough capacity, so they cram information via superposition—using each neuron for many unrelated things (Chinese, fishing, horses)—which is why removing a single neuron yields no interpretable meaning; sparse autoencoders address this by projecting into a higher-dimensional space where concepts separate cleanly.Is RL + LLMs enough for AGI? — Sholto Douglas & Trenton Bricken
- Sparse autoencoders are like telescopes revealing features, but there is strong evidence we still see only a small fraction of them — a kind of 'dark matter' in the neural network universe that may never be observable or computationally tractable to observe, which has implications for safety if a significant fraction of networks remains inaccessible.Dario Amodei