Mathematical Reasoning Calibration on MATH 500
0.0725Brier ScoreOT
Evaluation Results
| Method | Links | ||||
|---|---|---|---|---|---|
| OTGenerator Model=Llama-3.2-1B, Scorer Model=Qwen2.5-Math-PRM-7B, Calibration Variant=OT2026.05 | 0.0725 | 0.0425 | 0.0772 | 0.0608 | |
| OTGenerator Model=Qwen-2.5-7B, Scorer Model=Qwen2.5-Math-PRM-7B, Calibration Variant=OT2026.05 | 0.0841 | 0.0639 | 0.0275 | 0.0677 | |
| Shepherd-7B (Llama-3.2-1B)Backbone=Llama-3.2-1B, Calibration=OT2026.05 | 0.0996 | 0.0653 | 0.1125 | 0.0713 | |
| Shepherd-7B (Qwen-2.5-7B)Backbone=Qwen-2.5-7B, Calibration=OT2026.05 | 0.1079 | 0.083 | 0.0505 | 0.0818 | |
| Shepherd-7B (Qwen-2.5-7B)Backbone=Qwen-2.5-7B, Calibration=Base2026.05 | 0.1136 | 0.0865 | 0.0348 | — | |
| ReasonEval-7BBackbone=Llama-3.2-1B, Protocol=OT2026.05 | 0.119 | 0.0784 | 0.1446 | 0.077 | |
| OTGenerator Model=R1-Qwen-7B, Scorer Model=Qwen2.5-Math-PRM-7B, Calibration Variant=OT2026.05 | 0.1226 | 0.0782 | 0.0309 | 0.0846 | |
| ReasonEval-7BBackbone=R1-Qwen-7B, Protocol=OT2026.05 | 0.1281 | 0.0524 | 0.113 | 0.0869 | |
| QRGenerator Model=R1-Llama-8B, Scorer Model=Qwen2.5-Math-PRM-7B, Calibration Variant=QR2026.05 | 0.1347 | 0.0761 | 0.1082 | 0.1268 | |
| OTGenerator Model=R1-Llama-8B, Scorer Model=Qwen2.5-Math-PRM-7B, Calibration Variant=OT2026.05 | 0.1352 | 0.0979 | 0.0522 | 0.0971 | |
| QRGenerator Model=R1-Qwen-7B, Scorer Model=Qwen2.5-Math-PRM-7B, Calibration Variant=QR2026.05 | 0.1367 | 0.0711 | 0.0918 | 0.1268 | |
| Shepherd-7B (R1-Qwen-7B)Backbone=R1-Qwen-7B, Calibration=QR2026.05 | 0.1388 | 0.0724 | 0.0909 | 0.1217 | |
| ReasonEval-7BBackbone=R1-Qwen-7B, Protocol=QR2026.05 | 0.1389 | 0.0801 | 0.1253 | 0.1345 | |
| OTGenerator Model=Qwen-2.5-1.5B, Scorer Model=Qwen2.5-Math-PRM-7B, Calibration Variant=OT2026.05 | 0.1398 | 0.121 | 0.1311 | 0.0921 | |
| ReasonEval-7BBackbone=Qwen-2.5-7B, Protocol=QR2026.05 | 0.1416 | 0.0884 | 0.1428 | 0.1345 | |
| ReasonEval-7BBackbone=Qwen-2.5-7B, Protocol=Base2026.05 | 0.143 | 0.1215 | 0.0978 | — | |
| ReasonEval-7BBackbone=R1-Llama-8B, Protocol=QR2026.05 | 0.1437 | 0.0852 | 0.1369 | 0.1361 | |
| Shepherd-7B (R1-Qwen-7B)Backbone=R1-Qwen-7B, Calibration=Base2026.05 | 0.1444 | 0.0445 | 0.1419 | — | |
| BaseGenerator Model=Qwen-2.5-7B, Scorer Model=Qwen2.5-Math-PRM-7B, Calibration Variant=Base2026.05 | 0.1444 | 0.1412 | 0.1712 | — | |
| QRGenerator Model=Qwen-2.5-7B, Scorer Model=Qwen2.5-Math-PRM-7B, Calibration Variant=QR2026.05 | 0.1444 | 0.0738 | 0.0873 | 0.1228 | |
| Shepherd-7B (R1-Llama-8B)Backbone=R1-Llama-8B, Calibration=OT2026.05 | 0.1528 | 0.1029 | 0.0767 | 0.0947 | |
| Shepherd-7B (R1-Llama-8B)Backbone=R1-Llama-8B, Calibration=QR2026.05 | 0.1534 | 0.0845 | 0.0653 | 0.1224 | |
| ReasonEval-7BBackbone=R1-Llama-8B, Protocol=OT2026.05 | 0.157 | 0.0805 | 0.0879 | 0.0985 | |
| Shepherd-7B (R1-Llama-8B)Backbone=R1-Llama-8B, Calibration=Base2026.05 | 0.1586 | 0.0583 | 0.1158 | — | |
| Shepherd-7B (Qwen-2.5-7B)Backbone=Qwen-2.5-7B, Calibration=QR2026.05 | 0.1618 | 0.0877 | 0.0741 | 0.1225 | |
| Shepherd-7B (R1-Qwen-7B)Backbone=R1-Qwen-7B, Calibration=OT2026.05 | 0.1659 | 0.0969 | 0.1368 | 0.0983 | |
| Shepherd-7B (Qwen-2.5-1.5B)Backbone=Qwen-2.5-1.5B, Calibration=Base2026.05 | 0.1669 | 0.1433 | 0.1341 | — | |
| Shepherd-7B (Llama-3.1-8B)Backbone=Llama-3.1-8B, Calibration=OT2026.05 | 0.1697 | 0.1297 | 0.1661 | 0.1078 | |
| QRGenerator Model=Qwen-2.5-1.5B, Scorer Model=Qwen2.5-Math-PRM-7B, Calibration Variant=QR2026.05 | 0.1736 | 0.1224 | 0.0478 | 0.1379 | |
| BaseGenerator Model=R1-Qwen-7B, Scorer Model=Qwen2.5-Math-PRM-7B, Calibration Variant=Base2026.05 | 0.1738 | 0.1694 | 0.1881 | — | |
| ReasonEval-7BBackbone=R1-Qwen-7B, Protocol=Base2026.05 | 0.177 | 0.1679 | 0.1825 | — | |
| ReasonEval-7BBackbone=Qwen-2.5-1.5B, Protocol=OT2026.05 | 0.1865 | 0.151 | 0.1017 | 0.1169 | |
| ReasonEval-7BBackbone=Qwen-2.5-1.5B, Protocol=QR2026.05 | 0.1869 | 0.1414 | 0.0587 | 0.1504 | |
| ReasonEval-7BBackbone=R1-Llama-8B, Protocol=Base2026.05 | 0.1934 | 0.1832 | 0.1948 | — | |
| BaseGenerator Model=R1-Llama-8B, Scorer Model=Qwen2.5-Math-PRM-7B, Calibration Variant=Base2026.05 | 0.1935 | 0.1868 | 0.2112 | — | |
| Shepherd-7B (Qwen-2.5-1.5B)Backbone=Qwen-2.5-1.5B, Calibration=QR2026.05 | 0.195 | 0.1327 | 0.0747 | 0.1361 | |
| Shepherd-7B (Llama-3.1-8B)Backbone=Llama-3.1-8B, Calibration=Base2026.05 | 0.1998 | 0.176 | 0.2458 | — | |
| OTGenerator Model=Llama-3.1-8B, Scorer Model=Qwen2.5-Math-PRM-7B, Calibration Variant=OT2026.05 | 0.2009 | 0.185 | 0.2406 | 0.1236 | |
| Shepherd-7B (Qwen-2.5-1.5B)Backbone=Qwen-2.5-1.5B, Calibration=OT2026.05 | 0.2127 | 0.1977 | 0.1853 | 0.1313 | |
| Shepherd-7B (Llama-3.2-1B)Backbone=Llama-3.2-1B, Calibration=Base2026.05 | 0.2137 | 0.2033 | 0.3422 | — | |
| ReasonEval-7BBackbone=Qwen-2.5-7B, Protocol=OT2026.05 | 0.2161 | 0.1268 | 0.1989 | 0.1183 | |
| QRGenerator Model=Llama-3.1-8B, Scorer Model=Qwen2.5-Math-PRM-7B, Calibration Variant=QR2026.05 | 0.2165 | 0.1873 | 0.2414 | 0.1609 | |
| ReasonEval-7BBackbone=Llama-3.1-8B, Protocol=OT2026.05 | 0.2181 | 0.1753 | 0.2012 | 0.1244 | |
| ReasonEval-7BBackbone=Qwen-2.5-1.5B, Protocol=Base2026.05 | 0.2225 | 0.202 | 0.2077 | — | |
| ReasonEval-7BBackbone=Llama-3.1-8B, Protocol=QR2026.05 | 0.2237 | 0.1983 | 0.2571 | 0.173 | |
| ReasonEval-7BBackbone=Llama-3.2-1B, Protocol=QR2026.05 | 0.2301 | 0.2159 | 0.3783 | 0.1905 | |
| QRGenerator Model=Llama-3.2-1B, Scorer Model=Qwen2.5-Math-PRM-7B, Calibration Variant=QR2026.05 | 0.2313 | 0.2188 | 0.3836 | 0.1774 | |
| Shepherd-7B (Llama-3.2-1B)Backbone=Llama-3.2-1B, Calibration=QR2026.05 | 0.2407 | 0.2235 | 0.3737 | 0.1721 | |
| BaseGenerator Model=Qwen-2.5-1.5B, Scorer Model=Qwen2.5-Math-PRM-7B, Calibration Variant=Base2026.05 | 0.2446 | 0.2387 | 0.2827 | — | |
| Shepherd-7B (Llama-3.1-8B)Backbone=Llama-3.1-8B, Calibration=QR2026.05 | 0.2601 | 0.2244 | 0.2672 | 0.1585 | |
| ReasonEval-7BBackbone=Llama-3.1-8B, Protocol=Base2026.05 | 0.3084 | 0.2894 | 0.3571 | — | |
| BaseGenerator Model=Llama-3.1-8B, Scorer Model=Qwen2.5-Math-PRM-7B, Calibration Variant=Base2026.05 | 0.3229 | 0.309 | 0.3878 | — | |
| BaseGenerator Model=Llama-3.2-1B, Scorer Model=Qwen2.5-Math-PRM-7B, Calibration Variant=Base2026.05 | 0.3524 | 0.3423 | 0.4402 | — | |
| ReasonEval-7BBackbone=Llama-3.2-1B, Protocol=Base2026.05 | 0.3629 | 0.3548 | 0.4804 | — |