# When LLM Judge Scores Look Good but Best-of-N Decisions Fail

Source: [When LLM Judge Scores Look Good but Best-of-N Decisions Fail](<https://www.sota2.com/research/paper/69b78e9e0636e5bf4afbdcc9>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-03-17T05:00:04.157Z

Authors: Eddie Landesberg

Published: 2026-03-12T23:40:03.000Z

## Abstract

Large language models are often used as judges to score candidate responses, then validated with a single global metric such as correlation with reference labels. This can be misleading when the real deployment task is best-of-n selection within a prompt.   In a 5,000-prompt best-of-4 benchmark from Chatbot Arena, a judge with moderate global correlation (r = 0.47) captures only 21.0% of the improvement that perfect selection would achieve over random choice. The gap arises because global agreement is driven largely by prompt-level baseline effects, while selection depends on within-prompt ranking: within-prompt correlation is only r\_within = 0.27, and coarse pointwise scoring creates ties in 67% of pairwise comparisons.   In a matched-pair best-of-2 audit, explicit pairwise judging recovers much of this lost signal, raising recovery from 21.1% to 61.2%. For judge-based selection, the relevant audit should report within-prompt signal, tie rates, and recovery/top-1 accuracy, not global agreement alone.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Multi-judge evaluation on Shared 500-prompt sample](<https://www.sota2.com/research/sota/multi-judge-evaluation-on-shared-500-prompt-sample.md>): Multi-judge evaluation; Shared 500-prompt sample; Global Correlation (r): not recorded.

- [LLM Judge Evaluation on LLM-to-LLM Evaluation Reference: GPT-5.2](<https://www.sota2.com/research/sota/llm-judge-evaluation-on-llm-to-llm-evaluation-reference-gpt-5-2.md>): LLM-judge evaluation; LLM-to-LLM Evaluation Reference: GPT-5.2; Global Correlation (r): not recorded.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2603.12520>)

[PDF](<https://arxiv.org/pdf/2603.12520.pdf>)
