How Better Feedback Can Revolutionize Education (with Daisy Christodoulou)
What this covers
Russ Roberts interviews educational consultant Daisy Christodoulou on how feedback systems in schools fail to drive improvement. The core argument is structural: written prose feedback—grades, comments on papers—does not work because it is not designed for action. Christodoulou contends that effective feedback must function like a thermostat, actively moving a student toward a defined goal rather than merely measuring where they stand. She explains why this matters by grounding the problem in how humans actually learn: tacit knowledge cannot be conveyed in words, improvement demands a model of progression with intermediate steps that look nothing like the final task, and working memory bottlenecks mean students cannot skip the hard work of building knowledge through structured practice.
The conversation ranges across the science of learning, the limits of artificial intelligence, and what teachers should actually do. Roberts and Christodoulou examine why comparative judgment—asking humans which of two essays is better—proves more reliable than trying to score a single piece against a rubric, and why large language models create new problems: they hallucinate, show position bias, and make errors that institutional feedback systems have no way to catch. Throughout, Christodoulou argues that vocabulary size and memorization deserve far more weight in education than they receive, that most of writing is teachable algorithm rather than magic, and that human in the loop remains essential—AI can handle transcription and combination, but judgment and the emotional scaffolding of feedback must stay with people. The conversation includes pushback on whether the model works as claimed and what gets lost when prose is sidelined.
Christodoulou argues that written prose feedback (grades and comments) is fundamentally not optimized for improvement; effective feedback must be a 'thermostat' that provides actionable steps within a structured model of progression—steps that often look nothing like the end goal—and that AI can help scale this only when humans remain strategically in the loop.
- Prose is not optimized for action because much skill is tacit knowledge (Polanyi) that cannot be conveyed in words
- Improvement requires a model of progression with intermediate activities that don't resemble the final task, as in marathon training
- LLMs hallucinate and show position bias, so they must be used for tasks they do well (transcribing, combining) while humans supply judgment and feedback
Learning progresses through structured intermediate activities that differ from the final task, not through repeated attempts at the goal itself.
- Just as effective marathon training does not consist of running marathons repeatedly but builds a graded plan including non-running activities like strength work and yoga, effective academic improvement requires a model of progression with intermediate activities that don't resemble the final task—true for novices and elites alike.
“Do you go out and run a marathon in every single training practice session? No, you do not.”
- The actions needed to move from a student's actual state to their goal state often look very different from either state; for example, to infer more insightfully a student needs wider vocabulary and more background knowledge, which is best built through lessons that involve no essay writing at all.
“the actions you need to take to get from your actual state to your goal state can often look very different to either the actual state or the goal state”
- Pushing students to write too much too soon before they have built the foundational building blocks causes them to embed bad habits through repetition and practice, analogous to attempting a marathon before adequate training.
“if you try to get students to write too much too soon when they haven't built the building blocks. Because, they will build bad habits, but they will embed those bad habits through repetition and through practice”
- Learning by doing must be heavily structured in early stages—the doing need not look like the end goal, and learners need guardrails (like a parent steadying a bike) to avoid rabbit holes and dead ends, with structure relaxed only as competence grows.
“you learn by doing, but you learn by a kind of heavily structured--to begin with--a very heavily structured method that is avoiding you going off onto rabbit holes and dead ends”
Effective feedback must actively move students toward a goal state, not merely describe deficiency or restate mark-scheme language without actionable steps.
- Feedback that restates the mark scheme's language back to the student—such as 'make your inquiries more systematic' or 'infer more insightfully'—is True But Useless (TBU): it accurately describes the deficiency but provides no actionable step, like telling an unsuccessful comedian to tell funnier jokes.
“this is what Dylan Wiliam would say is True But Useless. It is TBU. It's like telling an unsuccessful comedian to tell funnier jokes.”
- Educational feedback should function like a thermostat rather than a thermometer: a thermometer merely measures the student's current state (e.g., a grade), but effective feedback must provide information that actively moves the student closer to a defined goal state, closing the gap between actual and desired performance.
“a thermometer is a measurement tool: it will measure the temperature. A thermostat will change the temperature based on the thermometer's reading.”
- Written prose feedback is not an effective way of giving feedback in any category—whether technical accuracy, factual correctness, or higher-order analysis—because prose is not optimized for action or improvement.
“written feedback is not optimal for any of these categories, and that written feedback is not a particularly effective way of doing feedback”
- The most useful output of the audio-comment system is not the student's paragraph but a categorized feedback report for the teacher summarizing recurring themes across the whole class, since this functions as the 'thermostat' that lets the teacher re-plan and adjust their teaching sequence to address common issues.
“we can put together a feedback report for the teacher. So, this will be a summary of all of the audio comments, of all of the students in your class, and it will categorize them”
Large language models hallucinate unpredictably and display alien error patterns; mitigation requires keeping humans in the loop rather than automation or expecting near-term reliability.
- Large language models' propensity to hallucinate—confidently producing random, dangerous errors—is a huge problem for automation because software systems depend on determinism: you press button X and Y reliably happens, whereas with LLMs you press X and 95% of the time get Y but 5% get something wrong, and you cannot debug it because the model is a black box.
“what they do instead is they spit out these random, quite dangerous errors that they're incredibly confident about”
- LLMs combine the worst of both worlds for institutional systems: like humans they are inconsistent (unlike old deterministic algorithms), but unlike humans the kinds of errors they make (e.g., position bias) are alien, so existing systems built to catch and remediate typical human errors cannot handle them—and there is no obvious sanction when a model errs the way there is for a human.
“in one sense, they are more like humans: so they are inconsistent. But then in another sense, they are not like humans because the type of errors they make are not like human errors.”
- Per the Paradox of Automation, when a system handles 90-95% of cases but errs on 5-10%, asking humans to stay alert to catch those errors produces worse outcomes than having them do the whole task, because humans disengage—as with self-driving cars where a texting driver cannot re-attune fast enough to take over safely.
“when you have a system that is 90, 95% okay, but there's 5-to-10% of errors, actually asking humans to stay switched on to spot the 5- 10% of errors... you would get better outputs overall if they just did the whole thing themselves”
- Hallucinations are likely more of a feature than a bug of large language models and are not going to be solved anytime soon, so rather than ignoring them one should design systems that mitigate them—particularly by exploiting falling costs and keeping a human in the loop.
“We actually don't think that, I think they're a bit more of a feature than a bug.”
Writing skill comprises teachable algorithms and disciplines rather than pure talent, and foundational knowledge like vocabulary and subject understanding is crucial.
- Contrary to the view that there is no algorithm for becoming a good writer, roughly 90-99% of what it takes to be a good writer consists of teachable algorithms and disciplines (such as identifying the subject and verb in every sentence), with only a final 5% of magic inspiration separating the good from the superb.
“I probably think 90, 95% of what it takes to be a good writer is--there are some algorithms that are teachable; and then there's probably some magic inspirational last mile 5%”
- Creativity and the ability to make unusual connections depend on first committing substantial domain knowledge to memory; skills like creativity and communication are not abstract but tied to specific bodies of domain knowledge, so memorization is a crucial and underrated part of education.
“people often talk about creativity as being about making connections in unusual ways, but to make those connections, you have to know some stuff to begin with to connect between them”
- Vocabulary size correlates with and is probably causative for many other writing abilities, and limited vocabulary is one of the biggest factors holding back students as writers—including some older students and post-graduates—yet it is routinely taken for granted.
“vocabulary size correlates and is probably causative for so many other things. And there are so many students who simply--one of the things that is holding them back as writers is vocabulary”
Learning depends on working memory bottlenecks and long-term memory storage; tacit knowledge requires guided practice, and memorization cannot be outsourced to external tools.
- Per Polanyi's theory of tacit knowledge, there are many things we can do but cannot tell—intuitive understanding that is very hard to put into words, such as riding a bike or floating while swimming—which is why reading instructions is far less effective than guided practice.
“Michael Polanyi's entire theory of tacit knowledge is that there are things that we can do but we cannot tell.”
- The science of learning provides a rationale for breaking skills down and emphasizing memorization: we have a very limited working memory that bottlenecks how much information we can process, and a vast long-term memory whose stored knowledge we use to overcome that bottleneck—so you cannot outsource memory to Google or LLMs.
“we have a very limited working memory, which is a limit to the amount of information we can process in the environment. And, we have this vast long-term memory”
- The most effective revision strategy is self-testing—putting away notes and quizzing yourself with flashcards, fill-in-the-blanks, recalling equations—yet students' default and least effective strategy is rereading and highlighting their notes, sometimes highlighting nearly everything.
“their Number One go-to strategy is the least effective strategy: which is they will reread their notes and highlight them”
Humans excel at comparative judgment over absolute judgment; comparative pairwise assessment creates reliable measurement scales superior to essay-grading mark schemes.
- Humans are much better at comparative judgment (choosing which of two essays is better) than at absolute judgment (placing one essay on a mark scheme); collecting hundreds of thousands of such pairwise judgments and combining them algorithmically creates a reliable measurement scale, and human comparative judgment is the gold standard used to train LLMs.
“comparative judgment--humans are much better at comparative judgment than at absolute judgment”
- Unlike humans, who across roughly 40 million judgments show no significant position bias (about 50.5% left vs 49.5% right), large language models display strong position bias of 10-25%, frequently changing which essay they prefer when the same two pieces are presented in reversed left-right order, typically favoring the left.
“humans do not display significant position bias... something like 50.5% of human judgments are for the left and 49.5 are for the right”
Human judgment and connection cannot be fully replaced by technology; remote learning during COVID demonstrated children's education requires in-person human stability.
- Instead of having AI generate feedback or having teachers edit AI feedback, the better workflow is to have teachers leave blunt audio comments while judging, then use AI only to transcribe and combine those human comments into softened written feedback—using AI for tasks it does well (transcribing, combining) while keeping the substance human.
“they will leave an audio comment on the piece of writing... we use the AI in two ways. The first thing to transcribe the audio comments; and the second is to combine all of those audio comments into some written feedback”
- Although prose feedback is suboptimal for improvement, it has genuine value in making students feel seen and motivated; students put more effort into work they know a respected teacher reads, which is why teachers resist pure direct-AI feedback, and the audio system preserves this by telling students their feedback came from several real teachers who picked out phrases they loved.
“the kids like it. It motivates the students. They want to feel seen.”
- COVID lockdowns provided a near-perfect global experiment in whether children can learn what they need by sitting in front of a screen all day; the result—that in most countries people urgently wanted schools to reopen because students weren't learning enough remotely—vindicates the view that young children's education requires human stability and connection that technology cannot substitute.
“if it was possible for a seven-year-old to learn what they needed at school--not at school, but by sitting in front of a screen--I think someone would have cracked it during COVID”
Expert performance depends on field feedback quality; language fluency should not be mistaken for accuracy since language evolved for storytelling rather than truth.
- Whether expert human judgment outperforms a simple algorithm depends on whether the expert's field provides good feedback: in fields with rich feedback, experts become true experts who make good decisions; in fields with poor feedback, experts make bad decisions and an algorithm is preferable.
“it's all to do actually with feedback. It's about what are the kinds of expert fields where you get really good feedback?”
- Language is not always optimized for truth and precision—its evolutionary origin may lie in telling flattering stories and lying about ourselves—and humans likely invented numbers and mathematics precisely to get around the limitations of language, which is why LLMs' fluency with language should not be mistaken for accuracy.
“Language is not always optimized for truth and precision... the evolutionary origin language is to kind of lie and tell nice stories about ourselves”
- Asking whether a 17-year-old today should still learn to drive is a useful thought experiment for putting a number on AI optimism: even people most bullish about LLMs fixing hallucinations answer yes, implicitly revealing they don't expect self-driving cars (or fully reliable AI) to arrive soon.
“A little experiment I like to play with people is: Would you recommend a 17-year-old today should learn to drive?”
- Falling LLM costs (sparked by cheaper models like DeepSeek) enable mitigation strategies previously prohibitive—such as sending every essay to be comparatively judged twice in each order, identifying the pieces where the model disagrees with itself, and directing human attention only to those harder, closer cases.
“what we can do now is we can send every piece off to be comparatively judged twice, each way around. And then we can identify the ones where it's disagreeing with itself.”
- Assessment is to education what the price function is to a market economy: not an oppressive imposition of scarcity but a marvel that measures wants and directs energy, attention, and innovation; so imagining an ideal education with no assessment is like imagining an ideal market system with no prices.
“Assessment is the price function of education. You can't just turn around and go, in an ideal world, my ideal education has no assessment. That's like saying, 'My ideal market-based system has no price.'”
AI tools pose greater risks for novices lacking background knowledge to interrogate answers, but have more legitimate applications for expert students.
- AI tools have more applications for older, more expert students who possess the background knowledge to interrogate answers and ask the right questions, and fewer (with greater risk) for younger, less expert students who can be led astray.
“they have more applications for the older, more-expert student, fewer for the younger”
- LLMs risk repeating the discredited 'you can just Google it' error in a worse form: novices lack the background knowledge to ask the right questions or interrogate answers, making LLM use a form of discovery learning, and unlike Google the answers aren't even reliable.
“it's almost a form of discovery learning, and novices in particular, are they going to be asking the right questions? Do they have enough background knowledge to interrogate the answer that they're getting?”