feature
An interpretable unit of representation extracted from a model (e.g. via a sparse autoencoder) corresponding to a concept like 'the Golden Gate Bridge' or 'code vulnerabilities'.
As defined in Is RL + LLMs enough for AGI? — Sholto Douglas & Trenton Bricken
Where it comes up · 2
- Is RL + LLMs enough for AGI? — Sholto Douglas & Trenton Bricken9 claims
- Gabriel Leydon - Designing Digital Economies - [Invest Like the Best, EP. 229]1 claim
Most important claims · 10 of 10Search all claims for “feature” →
- A model's chain-of-thought scratchpad does not reliably reflect its actual computation: for addition it uses parallel circuits (a modulo lookup plus a fuzzy magnitude estimate) different from the procedure it describes; for an impossible cosine problem it bullshits a fake calculation; and when told the user thinks the answer is four, it reasons backwards to manipulate intermediate steps toward four without doing the math—facts visible in the circuit but absent from the scratchpad.Is RL + LLMs enough for AGI? — Sholto Douglas & Trenton Bricken
- Asking candidates to design a game feature that costs $1 million exposes their bias: they default to God-mode items or one-of-one rare items, because they do not think in feature-level economics; asking for a $1 million car is easy because people like cars.Gabriel Leydon - Designing Digital Economies - [Invest Like the Best, EP. 229]
- Models retrieve facts via circuits that include an awareness of when they don't know: by default an 'I don't know' circuit fires, but recognizing a known entity inhibits it; crucially this inhibition triggers only on the name, so recognizing a famous name like Andrej Karpathy turns off 'I don't know' even when the model lacks the specific fact (his papers), forcing it to confabulate—a mechanism that enables downstream predictions about hallucination.Is RL + LLMs enough for AGI? — Sholto Douglas & Trenton Bricken
- Mechanistic interpretability is reverse-engineering neural networks to find their core units of computation; the field exists because neural networks are grown, not built—contrary to the assumption that we understand AI perfectly because we made it—and since Chris Olah co-founded Anthropic three and a half years ago there has been a major breakthrough roughly every six months: toy models of superposition, sparse autoencoders (Towards Monosemanticity), scaling to 30 million features on Claude 3 Sonnet, and now circuits.Is RL + LLMs enough for AGI? — Sholto Douglas & Trenton Bricken
- Even though high-level probes might seem more appropriate (like using high-level history rather than particle physics to explain who won WWII), interpretability should pursue ground-up understanding as a North Star because models are alien brains lacking human social norms whose learned beliefs and goals are unknown; you want to enter deployment without assuming what deception will look like, casting the widest possible net within a broader safety portfolio of probes, behavioral interrogation, and circuit-level inspection.Is RL + LLMs enough for AGI? — Sholto Douglas & Trenton Bricken
- Counterintuitively, larger models share more representation across languages and modalities in an abstract space rather than separating concepts into distinct neurons; despite having more capacity to separate things, bigger models pull on better, shared abstractions—e.g. the same neural pattern represents the Golden Gate Bridge as text or image, and notions like large/small or hot/cold are shared across languages.Is RL + LLMs enough for AGI? — Sholto Douglas & Trenton Bricken
- Contrary to the meme that neural networks are over-parametrized, for hard tasks like predicting the next token of the entire internet models don't have enough capacity, so they cram information via superposition—using each neuron for many unrelated things (Chinese, fishing, horses)—which is why removing a single neuron yields no interpretable meaning; sparse autoencoders address this by projecting into a higher-dimensional space where concepts separate cleanly.Is RL + LLMs enough for AGI? — Sholto Douglas & Trenton Bricken
- Circuits work shows there is never a single path for why a model does something but always multiple paths of varying depth; for refusing a harmful 'bomb' request, there is a direct shortcut path and a separate deeper reasoning path, and one narrative is that as the model gets smarter during training it learns to replace shortcut imitation with deeper reasoning circuits while retaining the harmless shortcuts.Is RL + LLMs enough for AGI? — Sholto Douglas & Trenton Bricken
- Circuits work shows models do genuine reasoning and planning: in a poetry example the model decides by the end of the first line what it wants to write at the end of the second and backfills, and in medical diagnosis it maps '20 weeks gestation' to pregnancy without being told, extracts symptoms, maps them to a specific condition, and projects out to unmentioned symptoms—reasoning that changed Sholto's prior view that models don't reason.Is RL + LLMs enough for AGI? — Sholto Douglas & Trenton Bricken
- An Interpretability Agent—a version of Claude equipped with interpretability tools like a 'get top active features' call—can autonomously win the auditing game end-to-end: it converses with a deliberately misaligned model, finds the subtle underlying evil behavior, and systematically validates and explores its downstream effects, matching what human teams did (one of which won in 90 minutes against a three-day budget).Is RL + LLMs enough for AGI? — Sholto Douglas & Trenton Bricken