List your company
ResearchFormerly WizWand

Mathematical Reasoning Calibration on MATH 500

0.0725Brier ScoreOT
0.0608840.1392920.21770.296108May 7, 2026
Updated 6mo ago

Evaluation results

MethodLinks
OTGenerator Model=Llama-3.2-1B, Scorer Model=Qwen2.5-Math-PRM-7B, Calibration Variant=OT2026.050.07250.04250.07720.0608
OTGenerator Model=Qwen-2.5-7B, Scorer Model=Qwen2.5-Math-PRM-7B, Calibration Variant=OT2026.050.08410.06390.02750.0677
Shepherd-7B (Llama-3.2-1B)Backbone=Llama-3.2-1B, Calibration=OT2026.050.09960.06530.11250.0713
Shepherd-7B (Qwen-2.5-7B)Backbone=Qwen-2.5-7B, Calibration=OT2026.050.10790.0830.05050.0818
Shepherd-7B (Qwen-2.5-7B)Backbone=Qwen-2.5-7B, Calibration=Base2026.050.11360.08650.0348—
ReasonEval-7BBackbone=Llama-3.2-1B, Protocol=OT2026.050.1190.07840.14460.077
OTGenerator Model=R1-Qwen-7B, Scorer Model=Qwen2.5-Math-PRM-7B, Calibration Variant=OT2026.050.12260.07820.03090.0846
ReasonEval-7BBackbone=R1-Qwen-7B, Protocol=OT2026.050.12810.05240.1130.0869
QRGenerator Model=R1-Llama-8B, Scorer Model=Qwen2.5-Math-PRM-7B, Calibration Variant=QR2026.050.13470.07610.10820.1268
OTGenerator Model=R1-Llama-8B, Scorer Model=Qwen2.5-Math-PRM-7B, Calibration Variant=OT2026.050.13520.09790.05220.0971
QRGenerator Model=R1-Qwen-7B, Scorer Model=Qwen2.5-Math-PRM-7B, Calibration Variant=QR2026.050.13670.07110.09180.1268
Shepherd-7B (R1-Qwen-7B)Backbone=R1-Qwen-7B, Calibration=QR2026.050.13880.07240.09090.1217
ReasonEval-7BBackbone=R1-Qwen-7B, Protocol=QR2026.050.13890.08010.12530.1345
OTGenerator Model=Qwen-2.5-1.5B, Scorer Model=Qwen2.5-Math-PRM-7B, Calibration Variant=OT2026.050.13980.1210.13110.0921
ReasonEval-7BBackbone=Qwen-2.5-7B, Protocol=QR2026.050.14160.08840.14280.1345
ReasonEval-7BBackbone=Qwen-2.5-7B, Protocol=Base2026.050.1430.12150.0978—
ReasonEval-7BBackbone=R1-Llama-8B, Protocol=QR2026.050.14370.08520.13690.1361
Shepherd-7B (R1-Qwen-7B)Backbone=R1-Qwen-7B, Calibration=Base2026.050.14440.04450.1419—
BaseGenerator Model=Qwen-2.5-7B, Scorer Model=Qwen2.5-Math-PRM-7B, Calibration Variant=Base2026.050.14440.14120.1712—
QRGenerator Model=Qwen-2.5-7B, Scorer Model=Qwen2.5-Math-PRM-7B, Calibration Variant=QR2026.050.14440.07380.08730.1228
Shepherd-7B (R1-Llama-8B)Backbone=R1-Llama-8B, Calibration=OT2026.050.15280.10290.07670.0947
Shepherd-7B (R1-Llama-8B)Backbone=R1-Llama-8B, Calibration=QR2026.050.15340.08450.06530.1224
ReasonEval-7BBackbone=R1-Llama-8B, Protocol=OT2026.050.1570.08050.08790.0985
Shepherd-7B (R1-Llama-8B)Backbone=R1-Llama-8B, Calibration=Base2026.050.15860.05830.1158—
Shepherd-7B (Qwen-2.5-7B)Backbone=Qwen-2.5-7B, Calibration=QR2026.050.16180.08770.07410.1225
Shepherd-7B (R1-Qwen-7B)Backbone=R1-Qwen-7B, Calibration=OT2026.050.16590.09690.13680.0983
Shepherd-7B (Qwen-2.5-1.5B)Backbone=Qwen-2.5-1.5B, Calibration=Base2026.050.16690.14330.1341—
Shepherd-7B (Llama-3.1-8B)Backbone=Llama-3.1-8B, Calibration=OT2026.050.16970.12970.16610.1078
QRGenerator Model=Qwen-2.5-1.5B, Scorer Model=Qwen2.5-Math-PRM-7B, Calibration Variant=QR2026.050.17360.12240.04780.1379
BaseGenerator Model=R1-Qwen-7B, Scorer Model=Qwen2.5-Math-PRM-7B, Calibration Variant=Base2026.050.17380.16940.1881—
ReasonEval-7BBackbone=R1-Qwen-7B, Protocol=Base2026.050.1770.16790.1825—
ReasonEval-7BBackbone=Qwen-2.5-1.5B, Protocol=OT2026.050.18650.1510.10170.1169
ReasonEval-7BBackbone=Qwen-2.5-1.5B, Protocol=QR2026.050.18690.14140.05870.1504
ReasonEval-7BBackbone=R1-Llama-8B, Protocol=Base2026.050.19340.18320.1948—
BaseGenerator Model=R1-Llama-8B, Scorer Model=Qwen2.5-Math-PRM-7B, Calibration Variant=Base2026.050.19350.18680.2112—
Shepherd-7B (Qwen-2.5-1.5B)Backbone=Qwen-2.5-1.5B, Calibration=QR2026.050.1950.13270.07470.1361
Shepherd-7B (Llama-3.1-8B)Backbone=Llama-3.1-8B, Calibration=Base2026.050.19980.1760.2458—
OTGenerator Model=Llama-3.1-8B, Scorer Model=Qwen2.5-Math-PRM-7B, Calibration Variant=OT2026.050.20090.1850.24060.1236
Shepherd-7B (Qwen-2.5-1.5B)Backbone=Qwen-2.5-1.5B, Calibration=OT2026.050.21270.19770.18530.1313
Shepherd-7B (Llama-3.2-1B)Backbone=Llama-3.2-1B, Calibration=Base2026.050.21370.20330.3422—
ReasonEval-7BBackbone=Qwen-2.5-7B, Protocol=OT2026.050.21610.12680.19890.1183
QRGenerator Model=Llama-3.1-8B, Scorer Model=Qwen2.5-Math-PRM-7B, Calibration Variant=QR2026.050.21650.18730.24140.1609
ReasonEval-7BBackbone=Llama-3.1-8B, Protocol=OT2026.050.21810.17530.20120.1244
ReasonEval-7BBackbone=Qwen-2.5-1.5B, Protocol=Base2026.050.22250.2020.2077—
ReasonEval-7BBackbone=Llama-3.1-8B, Protocol=QR2026.050.22370.19830.25710.173
ReasonEval-7BBackbone=Llama-3.2-1B, Protocol=QR2026.050.23010.21590.37830.1905
QRGenerator Model=Llama-3.2-1B, Scorer Model=Qwen2.5-Math-PRM-7B, Calibration Variant=QR2026.050.23130.21880.38360.1774
Shepherd-7B (Llama-3.2-1B)Backbone=Llama-3.2-1B, Calibration=QR2026.050.24070.22350.37370.1721
BaseGenerator Model=Qwen-2.5-1.5B, Scorer Model=Qwen2.5-Math-PRM-7B, Calibration Variant=Base2026.050.24460.23870.2827—
Shepherd-7B (Llama-3.1-8B)Backbone=Llama-3.1-8B, Calibration=QR2026.050.26010.22440.26720.1585
ReasonEval-7BBackbone=Llama-3.1-8B, Protocol=Base2026.050.30840.28940.3571—
BaseGenerator Model=Llama-3.1-8B, Scorer Model=Qwen2.5-Math-PRM-7B, Calibration Variant=Base2026.050.32290.3090.3878—
BaseGenerator Model=Llama-3.2-1B, Scorer Model=Qwen2.5-Math-PRM-7B, Calibration Variant=Base2026.050.35240.34230.4402—
ReasonEval-7BBackbone=Llama-3.2-1B, Protocol=Base2026.050.36290.35480.4804—