factual

Three Base64 features reveal alien representations

The model learned three distinct Base64 features—one firing for numbers, one for letters, and a third (initially mysterious) that fired only for the ASCII-decodable subset of Base64—demonstrating that models develop dense, alien representations of regions especially relevant to next-token prediction that no human would naturally carve out, which has implications for interpretability difficulty at scale.

factualpending

Speaker

Trenton Bricken

Evidence Quote

“The fact that the model learned these three different features and it took us a little while to figure out what was going on is very Shoggoth-esque.”

Source

Sholto Douglas & Trenton Bricken — How LLMs actually think— Dwarkesh Patel Podcast
Created: 6/13/2026, 3:31:11 AM

Your notes

Loading notes...