LLM-as-a-Judge on PreferenceBench
90.71AccuracyCalibraEval
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| CalibraEvalBase Model=Qwen-72B2024.10 | 90.71 | 0.69 | |
| CalibraEvalBase Model=GPT4o2024.10 | 90.49 | 4.29 | |
| PrideBase Model=GPT4o2024.10 | 90.47 | 5.24 | |
| DIBase Model=GPT4o2024.10 | 90.46 | 4.49 | |
| GPT4oCalibration=None2024.10 | 90.44 | 5.29 | |
| PrideBase Model=Qwen-72B2024.10 | 90.26 | 2.27 | |
| DIBase Model=Qwen-72B2024.10 | 90.21 | 0.8 | |
| Qwen-72BCalibration=None2024.10 | 90.12 | 2.54 | |
| CCBase Model=GPT4o2024.10 | 90.02 | 5.79 | |
| CCBase Model=Qwen-72B2024.10 | 89.3 | 5.41 | |
| DCBase Model=Qwen-72B2024.10 | 88.83 | 7.04 | |
| DCBase Model=GPT4o2024.10 | 88.22 | 5.99 | |
| GeniiBackbone=Qwen2.5-14B-Instruct, Evaluation Setup=Same Model Family2025.10 | 87.49 | — | |
| VanillaBackbone=Qwen2.5-14B-Instruct, Evaluation Setup=Same Model Family2025.10 | 86.64 | — | |
| CalibraEvalBase Model=ChatGPT2024.10 | 85.98 | 2.82 | |
| PrideBase Model=ChatGPT2024.10 | 85.68 | 3.51 | |
| ChatGPTCalibration=None2024.10 | 85.61 | 3.04 | |
| GeniiBackbone=Qwen2.5-7B-Instruct, Evaluation Setup=Different Model Families2025.10 | 85.34 | — | |
| CCBase Model=ChatGPT2024.10 | 84.83 | 3.13 | |
| CalibraEvalBase Model=Llama-3-8B2024.10 | 83.98 | 2.42 | |
| DIBase Model=ChatGPT2024.10 | 83.94 | 9.82 | |
| CalibraEvalBase Model=Qwen-14B2024.10 | 83.88 | 5.12 | |
| DCBase Model=Llama-3-8B2024.10 | 83.65 | 2.9 | |
| PrideBase Model=Qwen-14B2024.10 | 83.55 | 7.36 | |
| DIBase Model=Llama-3-8B2024.10 | 83.45 | 4.03 | |
| Llama-3-8BCalibration=None2024.10 | 83.43 | 3.36 | |
| PrideBase Model=Llama-3-8B2024.10 | 83.24 | 4.35 | |
| Self-ConsistencyBackbone=Qwen2.5-7B-Instruct, Evaluation Setup=Different Model Families2025.10 | 83 | — | |
| CCBase Model=Llama-3-8B2024.10 | 82.78 | 9.47 | |
| DCBase Model=ChatGPT2024.10 | 82.68 | 5.81 | |
| VanillaBackbone=Qwen2.5-7B-Instruct, Evaluation Setup=Different Model Families2025.10 | 82.43 | — | |
| VanillaBackbone=Qwen2.5-7B-Instruct, Evaluation Setup=Same Model Family2025.10 | 82.43 | — | |
| GeniiBackbone=Gemma2-9B-Instruct, Evaluation Setup=Different Model Families2025.10 | 82.13 | — | |
| Judgment SFTBackbone=Llama3.1-8B-Instruct, Evaluation Setup=Different Model Families2025.10 | 81.68 | — | |
| Qwen-14BCalibration=None2024.10 | 80.68 | 11.99 | |
| Self-ConsistencyBackbone=Gemma2-9B-Instruct, Evaluation Setup=Different Model Families2025.10 | 80.48 | — | |
| DCBase Model=Qwen-14B2024.10 | 79.96 | 10.07 | |
| Judgment SFTBackbone=Gemma2-9B-Instruct, Evaluation Setup=Different Model Families2025.10 | 79.83 | — | |
| CalibraEvalBase Model=Llama-3.1-8B2024.10 | 78.64 | 6.19 | |
| Judgment SFTBackbone=Qwen2.5-7B-Instruct, Evaluation Setup=Different Model Families2025.10 | 78.58 | — | |
| Long ReasoningBackbone=Llama3.1-8B-Instruct, Evaluation Setup=Different Model Families2025.10 | 78.53 | — | |
| VanillaBackbone=Gemma2-9B-Instruct, Evaluation Setup=Different Model Families2025.10 | 78.48 | — | |
| GeniiBackbone=Qwen2.5-7B-Instruct, Evaluation Setup=Same Model Family2025.10 | 78.46 | — | |
| CCBase Model=Qwen-14B2024.10 | 78.41 | 7.03 | |
| DCBase Model=Llama-3.1-8B2024.10 | 77.98 | 5.91 | |
| PrideBase Model=Llama-3.1-8B2024.10 | 77.92 | 9.47 | |
| CCBase Model=Llama-3.1-8B2024.10 | 77.91 | 6.31 | |
| Llama-3.1-8BCalibration=None2024.10 | 77.54 | 6.65 | |
| DIBase Model=Qwen-14B2024.10 | 76.14 | 15.77 | |
| Self-ConsistencyBackbone=Llama3.1-8B-Instruct, Evaluation Setup=Different Model Families2025.10 | 75.23 | — | |
| DIBase Model=Llama-3.1-8B2024.10 | 75.14 | 12.42 | |
| Long ReasoningBackbone=Qwen2.5-7B-Instruct, Evaluation Setup=Different Model Families2025.10 | 74.02 | — | |
| VanillaBackbone=Llama3.1-8B-Instruct, Evaluation Setup=Different Model Families2025.10 | 71.07 | — | |
| RepromptBackbone=Qwen2.5-7B-Instruct, Evaluation Setup=Different Model Families2025.10 | 70.82 | — | |
| RepromptBackbone=Gemma2-9B-Instruct, Evaluation Setup=Different Model Families2025.10 | 70.12 | — | |
| GeniiBackbone=Llama3.1-8B-Instruct, Evaluation Setup=Different Model Families2025.10 | 66.17 | — | |
| VanillaBackbone=Qwen2.5-3B-Instruct, Evaluation Setup=Same Model Family2025.10 | 66.07 | — | |
| GeniiBackbone=Qwen2.5-3B-Instruct, Evaluation Setup=Same Model Family2025.10 | 61.96 | — | |
| RepromptBackbone=Llama3.1-8B-Instruct, Evaluation Setup=Different Model Families2025.10 | 56.36 | — |