Known biases: position bias (preferring the first or last answer), verbosity bias (favoring longer answers), self-preference (a model favoring its own style), and sensitivity to rubric wording.
Mitigate: randomize or swap answer order and average, use clear scoring rubrics with examples, ask for reasoning before the score, and use a strong judge model that differs from the one being tested.
Calibrate against human labels on a sample, report agreement, and keep humans in the loop for high-stakes decisions.
Loading…