Correctness Calibration on PPE (Preference Policy Evaluation)
0.017KuiperProbe
Evaluation results
| Method | Links | ||
|---|---|---|---|
| ProbeModel Architecture=LLAMA 70B2025.12 | 0.017 | 0.025 | |
| ProbeModel Architecture=J1 LLAMA 70B2025.12 | 0.017 | 0.018 | |
| ProbeModel Architecture=QWEN 32B2025.12 | 0.02 | 0.028 | |
| ProbeModel Architecture=J1 QWEN 32B2025.12 | 0.029 | 0.039 | |
| ProbeModel Architecture=LLAMA 8B2025.12 | 0.047 | 0.075 | |
| ProbeModel Architecture=J1 LLAMA 8B2025.12 | 0.047 | 0.069 | |
| VerbalizedModel Architecture=J1 LLAMA 70B2025.12 | 0.076 | 0.074 | |
| MajorityModel Architecture=J1 QWEN 32B2025.12 | 0.085 | 0.109 | |
| ConsistencyModel Architecture=J1 QWEN 32B2025.12 | 0.098 | 0.13 | |
| VerbalizedModel Architecture=J1 QWEN 32B2025.12 | 0.103 | 0.105 | |
| VerbalizedModel Architecture=QWEN 32B2025.12 | 0.107 | 0.108 | |
| MajorityModel Architecture=QWEN 32B2025.12 | 0.125 | 0.137 | |
| ConsistencyModel Architecture=QWEN 32B2025.12 | 0.132 | 0.145 | |
| ConsistencyModel Architecture=J1 LLAMA 70B2025.12 | 0.134 | 0.144 | |
| VerbalizedModel Architecture=LLAMA 70B2025.12 | 0.135 | 0.131 | |
| MajorityModel Architecture=J1 LLAMA 70B2025.12 | 0.138 | 0.128 | |
| ConsistencyModel Architecture=LLAMA 8B2025.12 | 0.159 | 0.17 | |
| MajorityModel Architecture=LLAMA 8B2025.12 | 0.159 | 0.154 | |
| VerbalizedModel Architecture=LLAMA 8B2025.12 | 0.182 | 0.174 | |
| ConsistencyModel Architecture=LLAMA 70B2025.12 | 0.211 | 0.208 | |
| MajorityModel Architecture=LLAMA 70B2025.12 | 0.215 | 0.202 | |
| VerbalizedModel Architecture=J1 LLAMA 8B2025.12 | 0.219 | 0.213 | |
| ConsistencyModel Architecture=J1 LLAMA 8B2025.12 | 0.278 | 0.26 | |
| MajorityModel Architecture=J1 LLAMA 8B2025.12 | 0.279 | 0.261 |