Mathematical Reasoning Calibration on AIME 24-25
0.0181Brier ScoreOT
Evaluation Results
| Method | Links | ||||
|---|---|---|---|---|---|
| OTGenerator Model=Llama-3.2-1B, Scorer Model=Qwen2.5-Math-PRM-7B, Calibration Variant=OT2026.05 | 0.0181 | 0.0165 | 0.0883 | 0.0213 | |
| Shepherd-7B (Llama-3.2-1B)Backbone=Llama-3.2-1B, Calibration=OT2026.05 | 0.0408 | 0.0395 | 0.1484 | 0.039 | |
| ReasonEval-7BBackbone=Llama-3.2-1B, Protocol=OT2026.05 | 0.0607 | 0.0595 | 0.2084 | 0.0442 | |
| Shepherd-7B (Llama-3.2-1B)Backbone=Llama-3.2-1B, Calibration=Base2026.05 | 0.1256 | 0.1247 | 0.295 | — | |
| Shepherd-7B (R1-Qwen-7B)Backbone=R1-Qwen-7B, Calibration=Base2026.05 | 0.1717 | 0.1343 | 0.2204 | — | |
| Shepherd-7B (R1-Llama-8B)Backbone=R1-Llama-8B, Calibration=Base2026.05 | 0.1746 | 0.1363 | 0.2198 | — | |
| OTGenerator Model=Qwen-2.5-7B, Scorer Model=Qwen2.5-Math-PRM-7B, Calibration Variant=OT2026.05 | 0.1819 | 0.1783 | 0.3012 | 0.1275 | |
| Shepherd-7B (Llama-3.1-8B)Backbone=Llama-3.1-8B, Calibration=Base2026.05 | 0.184 | 0.1811 | 0.3711 | — | |
| Shepherd-7B (Llama-3.1-8B)Backbone=Llama-3.1-8B, Calibration=OT2026.05 | 0.2011 | 0.1988 | 0.4174 | 0.1209 | |
| OTGenerator Model=Qwen-2.5-1.5B, Scorer Model=Qwen2.5-Math-PRM-7B, Calibration Variant=OT2026.05 | 0.211 | 0.2048 | 0.3402 | 0.1346 | |
| ReasonEval-7BBackbone=R1-Qwen-7B, Protocol=OT2026.05 | 0.2185 | 0.1906 | 0.3076 | 0.14 | |
| OTGenerator Model=Llama-3.1-8B, Scorer Model=Qwen2.5-Math-PRM-7B, Calibration Variant=OT2026.05 | 0.2305 | 0.2293 | 0.4005 | 0.132 | |
| Shepherd-7B (Llama-3.2-1B)Backbone=Llama-3.2-1B, Calibration=QR2026.05 | 0.2421 | 0.2418 | 0.4782 | 0.1894 | |
| ReasonEval-7BBackbone=Llama-3.2-1B, Protocol=QR2026.05 | 0.2575 | 0.2571 | 0.5041 | 0.2167 | |
| Shepherd-7B (Qwen-2.5-7B)Backbone=Qwen-2.5-7B, Calibration=OT2026.05 | 0.2631 | 0.2576 | 0.4345 | 0.1776 | |
| QRGenerator Model=Llama-3.2-1B, Scorer Model=Qwen2.5-Math-PRM-7B, Calibration Variant=QR2026.05 | 0.2693 | 0.2689 | 0.5136 | 0.2023 | |
| QRGenerator Model=R1-Llama-8B, Scorer Model=Qwen2.5-Math-PRM-7B, Calibration Variant=QR2026.05 | 0.2746 | 0.2614 | 0.4159 | 0.1889 | |
| ReasonEval-7BBackbone=R1-Llama-8B, Protocol=OT2026.05 | 0.2793 | 0.2543 | 0.3825 | 0.1622 | |
| QRGenerator Model=Qwen-2.5-7B, Scorer Model=Qwen2.5-Math-PRM-7B, Calibration Variant=QR2026.05 | 0.2847 | 0.2785 | 0.4731 | 0.202 | |
| Shepherd-7B (Qwen-2.5-7B)Backbone=Qwen-2.5-7B, Calibration=Base2026.05 | 0.2878 | 0.2835 | 0.4533 | — | |
| OTGenerator Model=R1-Llama-8B, Scorer Model=Qwen2.5-Math-PRM-7B, Calibration Variant=OT2026.05 | 0.2918 | 0.2772 | 0.3964 | 0.1914 | |
| ReasonEval-7BBackbone=Llama-3.1-8B, Protocol=OT2026.05 | 0.2922 | 0.2905 | 0.5162 | 0.1501 | |
| OTGenerator Model=R1-Qwen-7B, Scorer Model=Qwen2.5-Math-PRM-7B, Calibration Variant=OT2026.05 | 0.2945 | 0.2823 | 0.4225 | 0.1839 | |
| QRGenerator Model=R1-Qwen-7B, Scorer Model=Qwen2.5-Math-PRM-7B, Calibration Variant=QR2026.05 | 0.3014 | 0.2866 | 0.4325 | 0.1956 | |
| Shepherd-7B (R1-Qwen-7B)Backbone=R1-Qwen-7B, Calibration=QR2026.05 | 0.3102 | 0.2975 | 0.444 | 0.1839 | |
| Shepherd-7B (R1-Llama-8B)Backbone=R1-Llama-8B, Calibration=QR2026.05 | 0.3116 | 0.2981 | 0.4418 | 0.1812 | |
| ReasonEval-7BBackbone=Llama-3.1-8B, Protocol=QR2026.05 | 0.3149 | 0.3135 | 0.5474 | 0.2274 | |
| QRGenerator Model=Llama-3.1-8B, Scorer Model=Qwen2.5-Math-PRM-7B, Calibration Variant=QR2026.05 | 0.3198 | 0.3186 | 0.5507 | 0.2148 | |
| ReasonEval-7BBackbone=R1-Llama-8B, Protocol=QR2026.05 | 0.3219 | 0.3104 | 0.4636 | 0.207 | |
| Shepherd-7B (Qwen-2.5-7B)Backbone=Qwen-2.5-7B, Calibration=QR2026.05 | 0.3266 | 0.3186 | 0.5034 | 0.2005 | |
| QRGenerator Model=Qwen-2.5-1.5B, Scorer Model=Qwen2.5-Math-PRM-7B, Calibration Variant=QR2026.05 | 0.334 | 0.3281 | 0.517 | 0.2086 | |
| Shepherd-7B (Qwen-2.5-1.5B)Backbone=Qwen-2.5-1.5B, Calibration=Base2026.05 | 0.3569 | 0.3522 | 0.5281 | — | |
| ReasonEval-7BBackbone=R1-Qwen-7B, Protocol=QR2026.05 | 0.3577 | 0.3463 | 0.4897 | 0.2169 | |
| ReasonEval-7BBackbone=Qwen-2.5-7B, Protocol=QR2026.05 | 0.3658 | 0.3612 | 0.5497 | 0.2184 | |
| ReasonEval-7BBackbone=Qwen-2.5-1.5B, Protocol=OT2026.05 | 0.3681 | 0.3599 | 0.5299 | 0.2146 | |
| Shepherd-7B (Llama-3.1-8B)Backbone=Llama-3.1-8B, Calibration=QR2026.05 | 0.3705 | 0.3692 | 0.5849 | 0.2117 | |
| Shepherd-7B (Qwen-2.5-1.5B)Backbone=Qwen-2.5-1.5B, Calibration=QR2026.05 | 0.3842 | 0.3791 | 0.5522 | 0.2113 | |
| ReasonEval-7BBackbone=Qwen-2.5-1.5B, Protocol=QR2026.05 | 0.4069 | 0.402 | 0.5824 | 0.2336 | |
| ReasonEval-7BBackbone=Llama-3.2-1B, Protocol=Base2026.05 | 0.4177 | 0.4171 | 0.5785 | — | |
| Shepherd-7B (R1-Llama-8B)Backbone=R1-Llama-8B, Calibration=OT2026.05 | 0.4283 | 0.4198 | 0.5435 | 0.217 | |
| BaseGenerator Model=Llama-3.2-1B, Scorer Model=Qwen2.5-Math-PRM-7B, Calibration Variant=Base2026.05 | 0.4488 | 0.4482 | 0.5745 | — | |
| ReasonEval-7BBackbone=Qwen-2.5-7B, Protocol=Base2026.05 | 0.4527 | 0.4495 | 0.5891 | — | |
| ReasonEval-7BBackbone=Llama-3.1-8B, Protocol=Base2026.05 | 0.4535 | 0.4515 | 0.602 | — | |
| Shepherd-7B (R1-Qwen-7B)Backbone=R1-Qwen-7B, Calibration=OT2026.05 | 0.4655 | 0.4518 | 0.5496 | 0.2374 | |
| BaseGenerator Model=Qwen-2.5-7B, Scorer Model=Qwen2.5-Math-PRM-7B, Calibration Variant=Base2026.05 | 0.4887 | 0.4882 | 0.564 | — | |
| BaseGenerator Model=Llama-3.1-8B, Scorer Model=Qwen2.5-Math-PRM-7B, Calibration Variant=Base2026.05 | 0.5055 | 0.5035 | 0.6166 | — | |
| ReasonEval-7BBackbone=Qwen-2.5-7B, Protocol=OT2026.05 | 0.5469 | 0.5423 | 0.6684 | 0.2698 | |
| ReasonEval-7BBackbone=Qwen-2.5-1.5B, Protocol=Base2026.05 | 0.55 | 0.547 | 0.667 | — | |
| Shepherd-7B (Qwen-2.5-1.5B)Backbone=Qwen-2.5-1.5B, Calibration=OT2026.05 | 0.5851 | 0.5841 | 0.7143 | 0.3175 | |
| BaseGenerator Model=Qwen-2.5-1.5B, Scorer Model=Qwen2.5-Math-PRM-7B, Calibration Variant=Base2026.05 | 0.5958 | 0.5948 | 0.6836 | — | |
| ReasonEval-7BBackbone=R1-Llama-8B, Protocol=Base2026.05 | 0.7016 | 0.7011 | 0.7654 | — | |
| BaseGenerator Model=R1-Llama-8B, Scorer Model=Qwen2.5-Math-PRM-7B, Calibration Variant=Base2026.05 | 0.7039 | 0.7031 | 0.7652 | — | |
| ReasonEval-7BBackbone=R1-Qwen-7B, Protocol=Base2026.05 | 0.7111 | 0.7106 | 0.767 | — | |
| BaseGenerator Model=R1-Qwen-7B, Scorer Model=Qwen2.5-Math-PRM-7B, Calibration Variant=Base2026.05 | 0.7158 | 0.7153 | 0.7703 | — |