LLM Judging Agreement on Reasoning Agreement Benchmark 500-sample human-annotated
0.91Cohen's KappaMulti-sub RM
Evaluation Results
| Method | Links | |
|---|---|---|
| Multi-sub RM2025.07 | 0.91 | |
| GPT-4o2025.07 | 0.9 | |
| Master-RM-7BModel Scale=7B2025.07 | 0.9 | |
| Qwen2.5-72B-InstructModel Scale=72B2025.07 | 0.88 | |
| Qwen2.5-32B-InstructModel Scale=32B2025.07 | 0.88 | |
| Qwen2.5-14B-InstructModel Scale=14B2025.07 | 0.88 | |
| Master-RM-32BModel Scale=32B2025.07 | 0.87 | |
| Qwen2.5-1.5B-InstructModel Scale=1.5B2025.07 | 0.83 | |
| Qwen2.5-3B-InstructModel Scale=3B2025.07 | 0.82 | |
| Omni-Judge2025.07 | 0.81 | |
| LLaMA3-70B-InstructModel Scale=70B2025.07 | 0.81 | |
| Qwen2.5-7B-InstructModel Scale=7B2025.07 | 0.8 | |
| LLaMA3-8B-InstructModel Scale=8B2025.07 | 0.73 | |
| General-Verifier2025.07 | 0.7 | |
| Qwen2.5-0.5B-InstructModel Scale=0.5B2025.07 | 0.1 |