normative
Grading experts should disagree somewhat
For subjective domains like poetry, you want some degree of consensus among exceptional graders that they are each doing a good job, but you do not want too much consensus, because disagreement captures edge-case model behaviors that deviate from the norm.
normativepending
Speaker
Brendan FoodyEvidence Quote
“you probably don’t want too much consensus because you also want to get all of these edge case scenarios of what are the models doing that might deviate a little bit from what the norm is”
Created: 6/12/2026, 4:53:54 PM
Your notes
Loading notes...