factual
Models are under-parametrized and use superposition
Contrary to the meme that neural networks are over-parametrized, for hard tasks like predicting the next token of the entire internet models don't have enough capacity, so they cram information via superposition—using each neuron for many unrelated things (Chinese, fishing, horses)—which is why removing a single neuron yields no interpretable meaning; sparse autoencoders address this by projecting into a higher-dimensional space where concepts separate cleanly.
factualpending
Speaker
Trenton BrickenEvidence Quote
“the way they learn to do that is to use each of their neurons, or units of computation in the model, for lots of different things... It'll fire for like Chinese and fishing and horses”
Created: 6/13/2026, 3:28:51 AM
Your notes
Loading notes...