# Inter-Rater Reliability (Kappa/ICC) on MT-Bench

Source: [Inter-Rater Reliability (Kappa/ICC) on MT-Bench](<https://www.sota2.com/research/sota/inter-rater-reliability-kappa-icc-on-mt-bench>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-02-26T05:47:48.150Z

Evaluation of the consistency and reliability of LLM-as-a-Judge evaluations on the MT-Bench dataset, measured using Kappa and Intraclass Correlation Coefficients (ICC).

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

| Scope | Recorded value |
| --- | --- |
| Task | LLM-as-a-Judge Evaluation Consistency |
| Dataset (including any stated split) | MTBench |
| Metric | Kappa |
| Unit | SCALAR |
| Direction | Higher is better |
| Results loaded | 6 of 6 |
| Coverage | All results currently returned for this benchmark |

Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol.

## Metrics

- [Kappa](<https://www.sota2.com/research/sota/inter-rater-reliability-kappa-icc-on-mt-bench.md?metric=Kappa>): Higher is better; unit: SCALAR.

- [ICC(2,k)](<https://www.sota2.com/research/sota/inter-rater-reliability-kappa-icc-on-mt-bench.md?metric=ICC(2%2Ck)>): Higher is better; unit: SCALAR.

- [ICC(3,k)](<https://www.sota2.com/research/sota/inter-rater-reliability-kappa-icc-on-mt-bench.md?metric=ICC(3%2Ck)>): Higher is better; unit: SCALAR.

## Recorded results

| Method | Kappa | Result date | Report paper |
| --- | --- | --- | --- |
| CalibraEval | 71.88 | Not recorded | 68fa75551e209131cea67449 |
| Qwen-72B (Default) | 71.35 | Not recorded | 68fa75551e209131cea67449 |
| CalibraEval | 28.63 | Not recorded | 68fa75551e209131cea67449 |
| Pride | 27.38 | Not recorded | 68fa75551e209131cea67449 |
| DI | 15.93 | Not recorded | 68fa75551e209131cea67449 |
| Llama-3-8B (Default) | 14.36 | Not recorded | 68fa75551e209131cea67449 |

### CalibraEval

Kappa: 71.88. Result ID: 69939b763a6c314849173609.

[Reporting paper](<https://www.sota2.com/research/paper/68fa75551e209131cea67449.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa75551e209131cea67449.md>)

[Implementation](<https://github.com/CSHaitao/CalibraEval>)

Recorded evaluation configuration: {"Base Model":"Qwen-72B"}. This record does not assert independent reproduction.

### Qwen-72B (Default)

Kappa: 71.35. Result ID: 69939b763a6c314849173608.

[Reporting paper](<https://www.sota2.com/research/paper/68fa75551e209131cea67449.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa70b91e209131cea4cdc5.md>)

Recorded evaluation configuration: {"Debiasing":"None"}. This record does not assert independent reproduction.

### CalibraEval

Kappa: 28.63. Result ID: 69939b763a6c314849173607.

[Reporting paper](<https://www.sota2.com/research/paper/68fa75551e209131cea67449.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa75551e209131cea67449.md>)

[Implementation](<https://github.com/CSHaitao/CalibraEval>)

Recorded evaluation configuration: {"Base Model":"Llama-3-8B"}. This record does not assert independent reproduction.

### Pride

Kappa: 27.38. Result ID: 69939b763a6c314849173606.

[Reporting paper](<https://www.sota2.com/research/paper/68fa75551e209131cea67449.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa724a1e209131cea55992.md>)

Recorded evaluation configuration: {"Base Model":"Llama-3-8B"}. This record does not assert independent reproduction.

### DI

Kappa: 15.93. Result ID: 69939b763a6c314849173605.

[Reporting paper](<https://www.sota2.com/research/paper/68fa75551e209131cea67449.md>)

Recorded evaluation configuration: {"Base Model":"Llama-3-8B"}. This record does not assert independent reproduction.

### Llama-3-8B (Default)

Kappa: 14.36. Result ID: 69939b763a6c314849173604.

[Reporting paper](<https://www.sota2.com/research/paper/68fa75551e209131cea67449.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa701f1e209131cea492fe.md>)

[Implementation](<https://github.com/facebookresearch/llama>)

Recorded evaluation configuration: {"Debiasing":"None"}. This record does not assert independent reproduction.

Page 1 of 1. This is an index page, not the full collection.