normative

Novel Problems Detect Real Understanding

To distinguish genuine understanding from memorization, you must give students or models a novel problem never asked before that requires first-principles reasoning; students who merely crammed and the LLM will both score zero, while the smart student who understood the concepts but didn't memorize will excel—even though testing this way requires the labor of inventing fresh problems each time.

normativepending

Speaker

François Chollet

Evidence Quote

“memorization is not a good proxy for understanding. Because you can always memorize your way into a high score with no understanding.”

Source

280 | François Chollet on Deep Learning and the Meaning of Intelligence— Sean Carroll's Mindscape
Created: 6/13/2026, 12:07:39 AM

Your notes

Loading notes...