factual
Attention approximates cerebellar circuit
Pentti Kanerva's 1980s associative memory algorithm, when implemented as an electrical circuit, looks identical to the core cerebellar circuit found in nearly every organism (including the Drosophila mushroom body), and Bricken's work shows the attention operation is a close approximation of this same circuit—a three-way convergence suggesting attention's success is not accidental.
factualpending
Speaker
Trenton BrickenEvidence Quote
“this attention operation is a very close approximation, including implementing the Softmax and having these nominal quadratic costs that we've been talking about. So the three way convergence here and the takeoff and success of transformers, just seems pretty striking to me.”
Created: 6/13/2026, 3:31:11 AM
Your notes
Loading notes...