# When Can You Debias an LLM Judge? Identifiability Limits, a Test, and Designs for Top-k Ranking

Source: [When Can You Debias an LLM Judge? Identifiability Limits, a Test, and Designs for Top-k Ranking](<https://www.sota2.com/research/paper/6a4741fb143ebdcacacc5b2e>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-07-22T05:00:05.107Z

Authors: Jian Xu, Delu Zeng, John Paisley, Qibin Zhao

Published: 2026-07-02T12:39:30.000Z

## Abstract

Large language models (LLMs) are increasingly used as cheap, scalable judges that compare candidate outputs pairwise. Because such judges prefer verbose or well-formatted answers, the natural fix is to add bias covariates to a Bradley--Terry model and estimate the bias away. We show this cannot work as advertised: the quality/bias split is \\emph{not identified} by pairwise comparisons, and the failure is exact -- across $48$ real judge-pools the profile likelihood over the coefficient is flat to $\\mathbf{0.0000}$ \\textbf{nats}, and scaling the comparisons $26\\times$ buys none. A \`\`debiased'' score is selected by the prior, not recovered from data. Our contribution is accordingly not a better estimator but a characterization of \\emph{when prior-based correction is justified}, plus designs that supply the missing information when it is not. The assumption the prior encodes -- quality is a priori uncorrelated with the covariate -- pays only while $\\mathrm{corr}(\\theta,x)$ stays below a crossing point (configuration-dependent, $0.22$--$0.60$), which is what makes the same model help on LLMBar and hurt on SummEval and Nectar. We give two escapes: a \\textbf{trusted-anchor gate} that decides per (judge, covariate, task) (no false enables in $6{,}000$ decisions at $K\\ge10$ anchors, a rate our sample bounds at $\\le6\\%$), and a \\textbf{paired rendering design}. Across fifteen real LLM judges bias is heterogeneous and capability-dependent: correction improves \\topk{} recall by $0.20$--$0.32$ on five biased-but-competent cheap judges and is a no-op on frontier ones (Spearman $\\rho{=}{-}0.84$ between competence and gain over the $14$ competent judges, $p{\<}10^{-3}$), concentrating the benefit where at-scale evaluation happens.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Top-k Ranking on SummEval](<https://www.sota2.com/research/sota/top-k-ranking-on-summeval.md>): Top-K Ranking; SummEval; Top-k Recall: 86.

- [Top-k Recall on Top-k Recall over oracles Synthetic Simulated](<https://www.sota2.com/research/sota/top-k-recall-on-top-k-recall-over-oracles-synthetic-simulated.md>): Top-k Recall; Top-k Recall over oracles Synthetic Simulated; Naive Top-k Recall: 84.

- [Top-k recovery on LLM Judge Comparison full-data](<https://www.sota2.com/research/sota/top-k-recovery-on-llm-judge-comparison-full-data.md>): Top-k recovery; LLM Judge Comparison full-data; Naive Recall: not recorded.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2607.02104>)

[PDF](<https://arxiv.org/pdf/2607.02104.pdf>)