factual

Models are under-parametrized and use superposition

Contrary to the meme that neural networks are over-parametrized, for hard tasks like predicting the next token of the entire internet models don't have enough capacity, so they cram information via superposition—using each neuron for many unrelated things (Chinese, fishing, horses)—which is why removing a single neuron yields no interpretable meaning; sparse autoencoders address this by projecting into a higher-dimensional space where concepts separate cleanly.

factualpending

Speaker

Trenton Bricken

Evidence Quote

“the way they learn to do that is to use each of their neurons, or units of computation in the model, for lots of different things... It'll fire for like Chinese and fishing and horses”

Source

Is RL + LLMs enough for AGI? — Sholto Douglas & Trenton Bricken— Dwarkesh Patel Podcast
Created: 6/13/2026, 3:28:51 AM

Your notes

Loading notes...