# Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge Source: [Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge]() Collection index: [llms.txt]() Data updated: 2025-11-13T05:01:22.196Z Authors: Lin Shi, Chiyu Ma, Wenhua Liang, Xingjian Diao, Weicheng Ma, Soroush Vosoughi Published: 2024-06-12T01:12:28.000Z ## Abstract LLM-as-a-Judge has emerged as a promising alternative to human evaluators across various tasks, yet inherent biases - particularly position bias, the tendency to favor solutions based on their position within the prompt - compromise its reliability. This exploratory study evaluates position bias in LLM judges across pairwise and list-wise comparison settings, introducing three metrics: repetition stability, position consistency, and preference fairness. Our experiments, involving 15 LLM judges across MTBench and DevBench with 22 tasks and approximately 40 solution-generating models, result in over 150,000 evaluation instances. We identify Judge-Level, Candidate-Level, and Task-Level factors contributing to bias. The findings confirm that position bias is not due to random chance and varies significantly across judges and tasks. While position bias is weakly influenced by the length of prompt components, it is strongly affected by the quality gap between solutions. Our agreement and disagreement analysis among judges further provides insights into the distribution of judging difficulty across the dataset, and highlights the potential for dataset modifications. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Open-ended Question Answering on ANTIQUE (S5)](): Open-ended Question Answering; ANTIQUE (S5); Kendall's Tau (K): 63.53. - [Open-ended QA Response Ranking on AlignBench Minos](): Open-ended QA Response Ranking; AlignBench Minos; K Score: 38.66. - [Open-ended QA Response Ranking on GaokaoBench Minos](): Open-ended QA Response Ranking; GaokaoBench Minos; K Score: 45.71. - [Open-ended Question Answering on TREC-DL-NF (S5)](): Open-ended Question Answering; TREC-DL-NF (S5); Kendall's Tau (K): 63.21. ## Original sources and implementation [Original paper]() [PDF]()