factual

Attention approximates cerebellar circuit

Pentti Kanerva's 1980s associative memory algorithm, when implemented as an electrical circuit, looks identical to the core cerebellar circuit found in nearly every organism (including the Drosophila mushroom body), and Bricken's work shows the attention operation is a close approximation of this same circuit—a three-way convergence suggesting attention's success is not accidental.

factualpending

Speaker

Trenton Bricken

Evidence Quote

“this attention operation is a very close approximation, including implementing the Softmax and having these nominal quadratic costs that we've been talking about. So the three way convergence here and the takeoff and success of transformers, just seems pretty striking to me.”

Source

Sholto Douglas & Trenton Bricken — How LLMs actually think— Dwarkesh Patel Podcast
Created: 6/13/2026, 3:31:11 AM

Your notes

Loading notes...