LLM Judge Consistency Evaluation on RewardBench
83.25KappaCalibraEval
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| CalibraEvalBase Model=GPT4o2024.10 | 83.25 | 96.25 | 96.27 | |
| CalibraEvalBase Model=Qwen-72B2024.10 | 82.8 | 95.47 | 95.75 | |
| GPT4o (Default)Debiasing=None2024.10 | 82.57 | 94.83 | 94.89 | |
| PrideBase Model=GPT4o2024.10 | 82.53 | 94.94 | 94.98 | |
| CCBase Model=GPT4o2024.10 | 81.38 | 94.47 | 94.48 | |
| PrideBase Model=Qwen-72B2024.10 | 78.64 | 92.99 | 93.3 | |
| DIBase Model=GPT4o2024.10 | 78.57 | 93.72 | 94.01 | |
| Qwen-72B (Default)Debiasing=None2024.10 | 78.28 | 92.77 | 93.13 | |
| DIBase Model=Qwen-72B2024.10 | 77.33 | 92.27 | 92.63 | |
| DCBase Model=GPT4o2024.10 | 76.89 | 92.28 | 92.35 | |
| CCBase Model=Qwen-72B2024.10 | 69.23 | 88.71 | 89.92 | |
| DCBase Model=Qwen-72B2024.10 | 66.61 | 87.03 | 87.9 | |
| CalibraEvalBase Model=ChatGPT2024.10 | 32.02 | 77.25 | 77.6 | |
| CalibraEvalBase Model=Llama-3-8B2024.10 | 30.32 | 86.51 | 86.66 | |
| PrideBase Model=ChatGPT2024.10 | 28.25 | 70.38 | 73.16 | |
| DCBase Model=ChatGPT2024.10 | 27.94 | 66.09 | 70.54 | |
| PrideBase Model=Llama-3-8B2024.10 | 27.65 | 72.72 | 73.77 | |
| CalibraEvalBase Model=Qwen-14B2024.10 | 26.4 | 75.75 | 76.11 | |
| PrideBase Model=Qwen-14B2024.10 | 24.73 | 67.46 | 68.3 | |
| DIBase Model=ChatGPT2024.10 | 24.52 | 70.23 | 71.58 | |
| CCBase Model=ChatGPT2024.10 | 24.28 | 57.25 | 58.23 | |
| DCBase Model=Llama-3-8B2024.10 | 23.57 | 69.04 | 72.79 | |
| DCBase Model=Llama-3.1-8B2024.10 | 23.17 | 72.89 | 76.22 | |
| DIBase Model=Llama-3.1-8B2024.10 | 21.59 | 74.04 | 80 | |
| Llama-3-8B (Default)Debiasing=None2024.10 | 20.81 | 66.24 | 71.79 | |
| CalibraEvalBase Model=Llama-3.1-8B2024.10 | 20.67 | 83.23 | 86.68 | |
| ChatGPT (Default)Debiasing=None2024.10 | 20.08 | 62.67 | 70.75 | |
| Qwen-14B (Default)Debiasing=None2024.10 | 19.69 | 63.41 | 66.86 | |
| DIBase Model=Llama-3-8B2024.10 | 19.63 | 64.98 | 70.87 | |
| CCBase Model=Llama-3.1-8B2024.10 | 16.89 | 59.24 | 60.09 | |
| DCBase Model=Qwen-14B2024.10 | 16.04 | 45.1 | 45.26 | |
| CCBase Model=Llama-3-8B2024.10 | 15.49 | 58.77 | 63.7 | |
| Llama-3.1-8B (Default)Debiasing=None2024.10 | 15.02 | 68.82 | 76.65 | |
| PrideBase Model=Llama-3.1-8B2024.10 | 14.63 | 66.62 | 76.02 | |
| DIBase Model=Qwen-14B2024.10 | 11.9 | 50.46 | 53.2 | |
| CCBase Model=Qwen-14B2024.10 | -1.62 | -4.71 | -4.93 |