# LLM-as-a-Judge Evaluation Consistency on PreferenceBench

Source: [LLM-as-a-Judge Evaluation Consistency on PreferenceBench](<https://www.sota2.com/research/sota/llm-as-a-judge-evaluation-consistency-on-preferencebench>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-02-26T05:47:48.163Z

Evaluation of the consistency and inter-rater reliability of LLM judges when scoring responses using the PreferenceBench dataset, measured using Kappa and Intraclass Correlation Coefficients (ICC).

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

| Scope | Recorded value |
| --- | --- |
| Task | LLM-as-a-Judge Evaluation Consistency |
| Dataset (including any stated split) | PreferenceBench |
| Metric | Kappa |
| Unit | PERCENT |
| Direction | Higher is better |
| Results loaded | 4 of 4 |
| Coverage | All results currently returned for this benchmark |

Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol.

## Metrics

- [Kappa](<https://www.sota2.com/research/sota/llm-as-a-judge-evaluation-consistency-on-preferencebench.md?metric=Kappa>): Higher is better; unit: PERCENT.

- [ICC(2,k)](<https://www.sota2.com/research/sota/llm-as-a-judge-evaluation-consistency-on-preferencebench.md?metric=ICC(2%2Ck)>): Higher is better; unit: PERCENT.

- [ICC(3,k)](<https://www.sota2.com/research/sota/llm-as-a-judge-evaluation-consistency-on-preferencebench.md?metric=ICC(3%2Ck)>): Higher is better; unit: PERCENT.

## Recorded results

| Method | Kappa | Result date | Report paper |
| --- | --- | --- | --- |
| CalibraEval | 79.73 | Not recorded | 68fa75551e209131cea67449 |
| GPT4o (Default) | 79.42 | Not recorded | 68fa75551e209131cea67449 |
| CalibraEval | 58.54 | Not recorded | 68fa75551e209131cea67449 |
| Llama-3-8B (Default) | 58.25 | Not recorded | 68fa75551e209131cea67449 |

### CalibraEval

Kappa: 79.73. Result ID: 69939b763a6c31484917360d.

[Reporting paper](<https://www.sota2.com/research/paper/68fa75551e209131cea67449.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa75551e209131cea67449.md>)

[Implementation](<https://github.com/CSHaitao/CalibraEval>)

Recorded evaluation configuration: {"Base Model":"GPT4o"}. This record does not assert independent reproduction.

### GPT4o (Default)

Kappa: 79.42. Result ID: 69939b763a6c31484917360c.

[Reporting paper](<https://www.sota2.com/research/paper/68fa75551e209131cea67449.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa727a1e209131cea56a1e.md>)

[Implementation](<https://github.com/openai/evals>)

Recorded evaluation configuration: {"Debiasing":"None"}. This record does not assert independent reproduction.

### CalibraEval

Kappa: 58.54. Result ID: 69939b763a6c31484917360b.

[Reporting paper](<https://www.sota2.com/research/paper/68fa75551e209131cea67449.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa75551e209131cea67449.md>)

[Implementation](<https://github.com/CSHaitao/CalibraEval>)

Recorded evaluation configuration: {"Base Model":"Llama-3-8B"}. This record does not assert independent reproduction.

### Llama-3-8B (Default)

Kappa: 58.25. Result ID: 69939b763a6c31484917360a.

[Reporting paper](<https://www.sota2.com/research/paper/68fa75551e209131cea67449.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa701f1e209131cea492fe.md>)

[Implementation](<https://github.com/facebookresearch/llama>)

Recorded evaluation configuration: {"Debiasing":"None"}. This record does not assert independent reproduction.

Page 1 of 1. This is an index page, not the full collection.