position bias
A systematic tendency to favor an option based on its presented position (e.g., left vs right) rather than its merits; present in LLMs at 10-25% but essentially absent in humans.
As defined in How Better Feedback Can Revolutionize Education (with Daisy Christodoulou)
Where it comes up · 1
Most important claims · 2 of 2Search all claims for “position bias” →
- LLMs combine the worst of both worlds for institutional systems: like humans they are inconsistent (unlike old deterministic algorithms), but unlike humans the kinds of errors they make (e.g., position bias) are alien, so existing systems built to catch and remediate typical human errors cannot handle them—and there is no obvious sanction when a model errs the way there is for a human.How Better Feedback Can Revolutionize Education (with Daisy Christodoulou)
- Unlike humans, who across roughly 40 million judgments show no significant position bias (about 50.5% left vs 49.5% right), large language models display strong position bias of 10-25%, frequently changing which essay they prefer when the same two pieces are presented in reversed left-right order, typically favoring the left.How Better Feedback Can Revolutionize Education (with Daisy Christodoulou)