# LLM Judgement Confidence Estimation on Chatbot Arena (test) Source: [LLM Judgement Confidence Estimation on Chatbot Arena (test)]() Collection index: [llms.txt]() Data updated: 2026-05-18T07:07:51.844Z Evaluation of LLMs on estimating the confidence associated with their judgments when acting as judges on the test set of the Chatbot Arena dataset, typically measured using Rank Correlation (RK) and AUROC. A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. | Scope | Recorded value | | --- | --- | | Task | LLM Judgement Confidence Estimation | | Dataset (including any stated split) | Chatbot Arena (test) | | Metric | RK | | Unit | SCALAR | | Direction | Higher is better | | Results loaded | 16 of 16 | | Coverage | All results currently returned for this benchmark | Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol. ## Metrics - [RK](): Higher is better; unit: SCALAR. - [AUROC](): Higher is better; unit: PERCENT. ## Recorded results | Method | RK | Result date | Report paper | | --- | --- | --- | --- | | Verbalized Confidence | 0.3418 | Not recorded | 6a0a9d11dbc836b87d82b3e2 | | Random Annotator | 0.3355 | Not recorded | 6a0a9d11dbc836b87d82b3e2 | | Predictive Probability | 0.3323 | Not recorded | 6a0a9d11dbc836b87d82b3e2 | | Simulated Annotators | 0.323 | Not recorded | 6a0a9d11dbc836b87d82b3e2 | | Learning Confidence (Vanilla) | 0.2817 | Not recorded | 6a0a9d11dbc836b87d82b3e2 | | Margin-Adaptive Confidence Ranking | 0.2743 | Not recorded | 6a0a9d11dbc836b87d82b3e2 | | Simulated Annotators | 0.2646 | Not recorded | 6a0a9d11dbc836b87d82b3e2 | | Random Annotator | 0.2629 | Not recorded | 6a0a9d11dbc836b87d82b3e2 | | Predictive Probability | 0.2552 | Not recorded | 6a0a9d11dbc836b87d82b3e2 | | Learning Confidence (Vanilla) | 0.2486 | Not recorded | 6a0a9d11dbc836b87d82b3e2 | | Random Annotator | 0.248 | Not recorded | 6a0a9d11dbc836b87d82b3e2 | | Simulated Annotators | 0.2469 | Not recorded | 6a0a9d11dbc836b87d82b3e2 | | Predictive Probability | 0.2457 | Not recorded | 6a0a9d11dbc836b87d82b3e2 | | Learning Confidence (Vanilla) | 0.2435 | Not recorded | 6a0a9d11dbc836b87d82b3e2 | | Margin-Adaptive Confidence Ranking | 0.2165 | Not recorded | 6a0a9d11dbc836b87d82b3e2 | | Margin-Adaptive Confidence Ranking | 0.2077 | Not recorded | 6a0a9d11dbc836b87d82b3e2 | ### Verbalized Confidence RK: 0.3418. Result ID: 6a0abaae47f34ce924f05dfe. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Judge Model":"Mistral-7B"}. This record does not assert independent reproduction. ### Random Annotator RK: 0.3355. Result ID: 6a0abaae47f34ce924f05dff. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Judge Model":"Mistral-7B"}. This record does not assert independent reproduction. ### Predictive Probability RK: 0.3323. Result ID: 6a0abaae47f34ce924f05dfd. [Reporting paper]() Recorded evaluation configuration: {"Judge Model":"Mistral-7B"}. This record does not assert independent reproduction. ### Simulated Annotators RK: 0.323. Result ID: 6a0abaae47f34ce924f05e00. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Judge Model":"Mistral-7B"}. This record does not assert independent reproduction. ### Learning Confidence (Vanilla) RK: 0.2817. Result ID: 6a0abaae47f34ce924f05e01. [Reporting paper]() Recorded evaluation configuration: {"Judge Model":"Mistral-7B"}. This record does not assert independent reproduction. ### Margin-Adaptive Confidence Ranking RK: 0.2743. Result ID: 6a0abaae47f34ce924f05e02. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Judge Model":"Mistral-7B"}. This record does not assert independent reproduction. ### Simulated Annotators RK: 0.2646. Result ID: 6a0abaae47f34ce924f05e05. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Judge Model":"Llama3-70B"}. This record does not assert independent reproduction. ### Random Annotator RK: 0.2629. Result ID: 6a0abaae47f34ce924f05e04. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Judge Model":"Llama3-70B"}. This record does not assert independent reproduction. ### Predictive Probability RK: 0.2552. Result ID: 6a0abaae47f34ce924f05e03. [Reporting paper]() Recorded evaluation configuration: {"Judge Model":"Llama3-70B"}. This record does not assert independent reproduction. ### Learning Confidence (Vanilla) RK: 0.2486. Result ID: 6a0abaae47f34ce924f05e06. [Reporting paper]() Recorded evaluation configuration: {"Judge Model":"Llama3-70B"}. This record does not assert independent reproduction. ### Random Annotator RK: 0.248. Result ID: 6a0abaae47f34ce924f05e09. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Judge Model":"Qwen2.5-72B"}. This record does not assert independent reproduction. ### Simulated Annotators RK: 0.2469. Result ID: 6a0abaae47f34ce924f05e0a. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Judge Model":"Qwen2.5-72B"}. This record does not assert independent reproduction. ### Predictive Probability RK: 0.2457. Result ID: 6a0abaae47f34ce924f05e08. [Reporting paper]() Recorded evaluation configuration: {"Judge Model":"Qwen2.5-72B"}. This record does not assert independent reproduction. ### Learning Confidence (Vanilla) RK: 0.2435. Result ID: 6a0abaae47f34ce924f05e0b. [Reporting paper]() Recorded evaluation configuration: {"Judge Model":"Qwen2.5-72B"}. This record does not assert independent reproduction. ### Margin-Adaptive Confidence Ranking RK: 0.2165. Result ID: 6a0abaae47f34ce924f05e07. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Judge Model":"Llama3-70B"}. This record does not assert independent reproduction. ### Margin-Adaptive Confidence Ranking RK: 0.2077. Result ID: 6a0abaae47f34ce924f05e0c. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Judge Model":"Qwen2.5-72B"}. This record does not assert independent reproduction. Page 1 of 1. This is an index page, not the full collection.