Confidence Calibration on ReClor (test)
4.4ECEORCE
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| ORCEFoundation Model=Qwen3 8B2026.05 | 4.4 | 0.626 | 8.051 | 0.064 | 0.041 | 79.4 | |
| Top-kFoundation Model=Qwen3 8B2026.05 | 4.6 | 0.133 | 2.797 | 0.161 | 0.139 | 80 | |
| ConfTunerFoundation Model=Qwen3 8B2026.05 | 4.7 | 0.351 | 6.047 | 0.091 | 0.069 | 79.4 | |
| ORCEFoundation Model=Mistral 7B2026.05 | 6.1 | 0.392 | 8.905 | 0.306 | 0.161 | 51.2 | |
| SFTFoundation Model=Qwen3 8B2026.05 | 6.2 | 0.537 | 9.431 | 0.083 | 0.06 | 79.4 | |
| ORCEFoundation Model=LLAMA-3 8B2026.05 | 6.5 | 0.544 | 8.061 | 0.169 | 0.08 | 60.8 | |
| SFTFoundation Model=Mistral 7B2026.05 | 7.3 | 0.344 | 2.368 | 0.329 | 0.184 | 51.2 | |
| SFTFoundation Model=LLAMA-3 8B2026.05 | 8.4 | 0.446 | 8.378 | 0.201 | 0.112 | 60.8 | |
| CoTFoundation Model=Qwen3 8B2026.05 | 12.4 | 0.084 | 0.062 | 0.183 | 0.161 | 79.6 | |
| ConfTunerFoundation Model=LLAMA-3 8B2026.05 | 14 | 0.343 | 3.196 | 0.25 | 0.161 | 60.8 | |
| ConfTunerFoundation Model=Mistral 7B2026.05 | 14.1 | 0.191 | 1.712 | 0.408 | 0.263 | 51.2 | |
| VanillaFoundation Model=Qwen3 8B2026.05 | 16.4 | 0.059 | 0.187 | 0.19 | 0.167 | 79.4 | |
| ConsistencyFoundation Model=Qwen3 8B2026.05 | 16.7 | 0.311 | 1.206 | 0.147 | 0.126 | 80.2 | |
| VanillaFoundation Model=LLAMA-3 8B2026.05 | 17.8 | 0.053 | 0.239 | 0.373 | 0.283 | 60.8 | |
| Top-kFoundation Model=LLAMA-3 8B2026.05 | 27 | 0.061 | 0.177 | 0.441 | 0.298 | 51.6 | |
| CoTFoundation Model=Mistral 7B2026.05 | 30 | 0.124 | 0.006 | 0.389 | 0.271 | 55.6 | |
| CoTFoundation Model=LLAMA-3 8B2026.05 | 33.8 | 0.067 | 0.134 | 0.441 | 0.304 | 52.4 | |
| Top-kFoundation Model=Mistral 7B2026.05 | 35.7 | 0.153 | 6.216 | 0.438 | 0.292 | 51 | |
| ConsistencyFoundation Model=LLAMA-3 8B2026.05 | 37.6 | 0.072 | 0.11 | 0.355 | 0.269 | 61.6 | |
| VanillaFoundation Model=Mistral 7B2026.05 | 46.8 | 0.049 | 0.275 | 0.477 | 0.332 | 51.2 | |
| ConsistencyFoundation Model=Mistral 7B2026.05 | 48.3 | 0.029 | 0.523 | 0.489 | 0.346 | 51.4 |