comparative judgment
An assessment method based on the finding that people judge more reliably by comparing two items ('which is better?') than by scoring one in isolation; many paired comparisons are combined by an algorithm to produce a measurement scale.
As defined in Coase, the Rules of the Game, and the Costs of Perfection (with Daisy Christodoulou)
Where it comes up · 2
- Coase, the Rules of the Game, and the Costs of Perfection (with Daisy Christodoulou)5 claims
- How Better Feedback Can Revolutionize Education (with Daisy Christodoulou)3 claims
Most important claims · 8 of 8Search all claims for “comparative judgment” →
- Humans are much better at comparative judgment (choosing which of two essays is better) than at absolute judgment (placing one essay on a mark scheme); collecting hundreds of thousands of such pairwise judgments and combining them algorithmically creates a reliable measurement scale, and human comparative judgment is the gold standard used to train LLMs.How Better Feedback Can Revolutionize Education (with Daisy Christodoulou)
- The core problem in traditional assessment is not disagreement between markers but disagreement within the same marker: in a study giving people 100 essays one week and 100 the next where six are secretly repeated, the same person gives the repeated essays different marks -- so the problem cannot be solved by hierarchy or an expert, because the expert will disagree with themselves.Coase, the Rules of the Game, and the Costs of Perfection (with Daisy Christodoulou)
- Human beings are poor at absolute judgments (e.g., 'how tall is that person?') but much better at comparative judgments (e.g., 'who is taller, left or right?'); aggregating many paired comparisons through an algorithm (Thurstone's law of comparative judgment) can build a reliable measurement scale.Coase, the Rules of the Game, and the Costs of Perfection (with Daisy Christodoulou)
- Paradoxically, having teachers make open-ended comparative judgments of student writing with no rubric produces very high agreement (objectivity), while marking the same essays with a detailed rubric and tick-lists produces low agreement (subjectivity), because comparative judgment captures tacit knowledge of quality that prose rubrics fail to specify.Coase, the Rules of the Game, and the Costs of Perfection (with Daisy Christodoulou)
- Comparative judgment is not mere rank-ordering; by linking large crowd-sourced assessments over time (e.g., 50,000 pieces of writing judged by thousands of teachers, each session linked to the last) it builds a robust distribution on which you can draw a line wherever your standards dictate, and it is especially good at holding that line consistently over time -- one of the hardest things to achieve in assessment.Coase, the Rules of the Game, and the Costs of Perfection (with Daisy Christodoulou)
- Unlike humans, who across roughly 40 million judgments show no significant position bias (about 50.5% left vs 49.5% right), large language models display strong position bias of 10-25%, frequently changing which essay they prefer when the same two pieces are presented in reversed left-right order, typically favoring the left.How Better Feedback Can Revolutionize Education (with Daisy Christodoulou)
- Falling LLM costs (sparked by cheaper models like DeepSeek) enable mitigation strategies previously prohibitive—such as sending every essay to be comparatively judged twice in each order, identifying the pieces where the model disagrees with itself, and directing human attention only to those harder, closer cases.How Better Feedback Can Revolutionize Education (with Daisy Christodoulou)
- Handball could be adjudicated by crowdsourcing comparative judgments of many video clips ('which is more of a handball?') from fans, players, and officials to build a distribution, then drawing a line on that distribution, and finally using AI pattern recognition to place a live incident above or below the line -- shifting from top-down decision review to bottom-up decision support.Coase, the Rules of the Game, and the Costs of Perfection (with Daisy Christodoulou)