normative
Novel Problems Detect Real Understanding
To distinguish genuine understanding from memorization, you must give students or models a novel problem never asked before that requires first-principles reasoning; students who merely crammed and the LLM will both score zero, while the smart student who understood the concepts but didn't memorize will excel—even though testing this way requires the labor of inventing fresh problems each time.
normativepending
Speaker
François CholletEvidence Quote
“memorization is not a good proxy for understanding. Because you can always memorize your way into a high score with no understanding.”
Source
280 | François Chollet on Deep Learning and the Meaning of Intelligence— Sean Carroll's MindscapeCreated: 6/13/2026, 12:07:39 AM
Your notes
Loading notes...