definition

Grokking Is Minimum Description Length

Grokking—the transition from a memorized, overfit input-output mapping to a generalizing solution—is an instance of the minimum description length principle: the program that generalizes best is the shortest, achieved by progressively pruning and compressing the mapping, so it is memorization plus regularization rather than a separate mechanism.

definitionpending

Speaker

François Chollet

Evidence Quote

“Grokking is a very, very old phenomenon. We've been observing it for decades. It's basically an instance of the minimum description length principle.”

Source

Francois Chollet — Why the biggest AI models can't solve simple puzzles— Dwarkesh Patel Podcast
Created: 6/13/2026, 3:26:16 AM

Your notes

Loading notes...