# SCOPE: Selective Conformal Optimized Pairwise LLM Judging

Source: [SCOPE: Selective Conformal Optimized Pairwise LLM Judging](<https://www.sota2.com/research/paper/6992a4bc9ebc499c97406dba>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-06-02T05:00:12.522Z

Authors: Sher Badshah, Ali Emami, Hassan Sajjad

Published: 2026-02-13T17:10:43.000Z

## Abstract

Large language models (LLMs) are increasingly used as scalable judges in pairwise evaluation, but they remain prone to miscalibration and biases. We propose SCOPE (Selective Conformal Optimized Pairwise Evaluation), a framework that calibrates an acceptance threshold so that, under exchangeability, the error rate among non-abstained judgments is at most a user-specified level $\\alpha$. To supply SCOPE with a bias-neutral uncertainty signal, we introduce Bidirectional Preference Entropy (BPE), which queries the judge under both response positions and converts the order-averaged preference probability into an entropy-based score. Across various pairwise judging benchmarks, BPE outperforms standard confidence proxies in calibration and discrimination, while SCOPE consistently satisfies the target risk bound (empirical FDR $\\approx 0.097$ to $0.099$ at $\\alpha = 0.10$) and retains substantial coverage. Compared to vanilla baselines, SCOPE accepts up to $2.4\\times$ more judgments under the same risk constraint, demonstrating that BPE enables reliable and high-coverage LLM-based evaluation.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Pairwise LLM Judging on RewardBench (Coverage and Risk)](<https://www.sota2.com/research/sota/pairwise-llm-judging-on-rewardbench-coverage-and-risk.md>): Pairwise LLM Judging; RewardBench; Coverage: 98.3.

- [Pairwise LLM Judging on Chatbot Arena](<https://www.sota2.com/research/sota/pairwise-llm-judging-on-chatbot-arena.md>): Pairwise LLM Judging; Chatbot Arena; Coverage: 58.3.

- [Pairwise LLM Judging on MT-Bench](<https://www.sota2.com/research/sota/pairwise-llm-judging-on-mt-bench.md>): Pairwise LLM Judging; MT-Bench; Coverage: 54.3.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2602.13110>)

[PDF](<https://arxiv.org/pdf/2602.13110.pdf>)