normative

Grading experts should disagree somewhat

For subjective domains like poetry, you want some degree of consensus among exceptional graders that they are each doing a good job, but you do not want too much consensus, because disagreement captures edge-case model behaviors that deviate from the norm.

normativepending

Speaker

Brendan Foody

Evidence Quote

“you probably don’t want too much consensus because you also want to get all of these edge case scenarios of what are the models doing that might deviate a little bit from what the norm is”

Source

Brendan Foody— Conversations with Tyler
Created: 6/12/2026, 4:53:54 PM

Your notes

Loading notes...