factual
Three Base64 features reveal alien representations
The model learned three distinct Base64 features—one firing for numbers, one for letters, and a third (initially mysterious) that fired only for the ASCII-decodable subset of Base64—demonstrating that models develop dense, alien representations of regions especially relevant to next-token prediction that no human would naturally carve out, which has implications for interpretability difficulty at scale.
factualpending
Speaker
Trenton BrickenEvidence Quote
“The fact that the model learned these three different features and it took us a little while to figure out what was going on is very Shoggoth-esque.”
Created: 6/13/2026, 3:31:11 AM
Your notes
Loading notes...