List your company
ResearchFormerly WizWand

Mathematical Reasoning Calibration on AIME 24-25

0.0181Brier ScoreOT
-0.0098080.1785710.366950.555329May 7, 2026
Updated 6mo ago

Evaluation results

MethodLinks
OTGenerator Model=Llama-3.2-1B, Scorer Model=Qwen2.5-Math-PRM-7B, Calibration Variant=OT2026.050.01810.01650.08830.0213
Shepherd-7B (Llama-3.2-1B)Backbone=Llama-3.2-1B, Calibration=OT2026.050.04080.03950.14840.039
ReasonEval-7BBackbone=Llama-3.2-1B, Protocol=OT2026.050.06070.05950.20840.0442
Shepherd-7B (Llama-3.2-1B)Backbone=Llama-3.2-1B, Calibration=Base2026.050.12560.12470.295—
Shepherd-7B (R1-Qwen-7B)Backbone=R1-Qwen-7B, Calibration=Base2026.050.17170.13430.2204—
Shepherd-7B (R1-Llama-8B)Backbone=R1-Llama-8B, Calibration=Base2026.050.17460.13630.2198—
OTGenerator Model=Qwen-2.5-7B, Scorer Model=Qwen2.5-Math-PRM-7B, Calibration Variant=OT2026.050.18190.17830.30120.1275
Shepherd-7B (Llama-3.1-8B)Backbone=Llama-3.1-8B, Calibration=Base2026.050.1840.18110.3711—
Shepherd-7B (Llama-3.1-8B)Backbone=Llama-3.1-8B, Calibration=OT2026.050.20110.19880.41740.1209
OTGenerator Model=Qwen-2.5-1.5B, Scorer Model=Qwen2.5-Math-PRM-7B, Calibration Variant=OT2026.050.2110.20480.34020.1346
ReasonEval-7BBackbone=R1-Qwen-7B, Protocol=OT2026.050.21850.19060.30760.14
OTGenerator Model=Llama-3.1-8B, Scorer Model=Qwen2.5-Math-PRM-7B, Calibration Variant=OT2026.050.23050.22930.40050.132
Shepherd-7B (Llama-3.2-1B)Backbone=Llama-3.2-1B, Calibration=QR2026.050.24210.24180.47820.1894
ReasonEval-7BBackbone=Llama-3.2-1B, Protocol=QR2026.050.25750.25710.50410.2167
Shepherd-7B (Qwen-2.5-7B)Backbone=Qwen-2.5-7B, Calibration=OT2026.050.26310.25760.43450.1776
QRGenerator Model=Llama-3.2-1B, Scorer Model=Qwen2.5-Math-PRM-7B, Calibration Variant=QR2026.050.26930.26890.51360.2023
QRGenerator Model=R1-Llama-8B, Scorer Model=Qwen2.5-Math-PRM-7B, Calibration Variant=QR2026.050.27460.26140.41590.1889
ReasonEval-7BBackbone=R1-Llama-8B, Protocol=OT2026.050.27930.25430.38250.1622
QRGenerator Model=Qwen-2.5-7B, Scorer Model=Qwen2.5-Math-PRM-7B, Calibration Variant=QR2026.050.28470.27850.47310.202
Shepherd-7B (Qwen-2.5-7B)Backbone=Qwen-2.5-7B, Calibration=Base2026.050.28780.28350.4533—
OTGenerator Model=R1-Llama-8B, Scorer Model=Qwen2.5-Math-PRM-7B, Calibration Variant=OT2026.050.29180.27720.39640.1914
ReasonEval-7BBackbone=Llama-3.1-8B, Protocol=OT2026.050.29220.29050.51620.1501
OTGenerator Model=R1-Qwen-7B, Scorer Model=Qwen2.5-Math-PRM-7B, Calibration Variant=OT2026.050.29450.28230.42250.1839
QRGenerator Model=R1-Qwen-7B, Scorer Model=Qwen2.5-Math-PRM-7B, Calibration Variant=QR2026.050.30140.28660.43250.1956
Shepherd-7B (R1-Qwen-7B)Backbone=R1-Qwen-7B, Calibration=QR2026.050.31020.29750.4440.1839
Shepherd-7B (R1-Llama-8B)Backbone=R1-Llama-8B, Calibration=QR2026.050.31160.29810.44180.1812
ReasonEval-7BBackbone=Llama-3.1-8B, Protocol=QR2026.050.31490.31350.54740.2274
QRGenerator Model=Llama-3.1-8B, Scorer Model=Qwen2.5-Math-PRM-7B, Calibration Variant=QR2026.050.31980.31860.55070.2148
ReasonEval-7BBackbone=R1-Llama-8B, Protocol=QR2026.050.32190.31040.46360.207
Shepherd-7B (Qwen-2.5-7B)Backbone=Qwen-2.5-7B, Calibration=QR2026.050.32660.31860.50340.2005
QRGenerator Model=Qwen-2.5-1.5B, Scorer Model=Qwen2.5-Math-PRM-7B, Calibration Variant=QR2026.050.3340.32810.5170.2086
Shepherd-7B (Qwen-2.5-1.5B)Backbone=Qwen-2.5-1.5B, Calibration=Base2026.050.35690.35220.5281—
ReasonEval-7BBackbone=R1-Qwen-7B, Protocol=QR2026.050.35770.34630.48970.2169
ReasonEval-7BBackbone=Qwen-2.5-7B, Protocol=QR2026.050.36580.36120.54970.2184
ReasonEval-7BBackbone=Qwen-2.5-1.5B, Protocol=OT2026.050.36810.35990.52990.2146
Shepherd-7B (Llama-3.1-8B)Backbone=Llama-3.1-8B, Calibration=QR2026.050.37050.36920.58490.2117
Shepherd-7B (Qwen-2.5-1.5B)Backbone=Qwen-2.5-1.5B, Calibration=QR2026.050.38420.37910.55220.2113
ReasonEval-7BBackbone=Qwen-2.5-1.5B, Protocol=QR2026.050.40690.4020.58240.2336
ReasonEval-7BBackbone=Llama-3.2-1B, Protocol=Base2026.050.41770.41710.5785—
Shepherd-7B (R1-Llama-8B)Backbone=R1-Llama-8B, Calibration=OT2026.050.42830.41980.54350.217
BaseGenerator Model=Llama-3.2-1B, Scorer Model=Qwen2.5-Math-PRM-7B, Calibration Variant=Base2026.050.44880.44820.5745—
ReasonEval-7BBackbone=Qwen-2.5-7B, Protocol=Base2026.050.45270.44950.5891—
ReasonEval-7BBackbone=Llama-3.1-8B, Protocol=Base2026.050.45350.45150.602—
Shepherd-7B (R1-Qwen-7B)Backbone=R1-Qwen-7B, Calibration=OT2026.050.46550.45180.54960.2374
BaseGenerator Model=Qwen-2.5-7B, Scorer Model=Qwen2.5-Math-PRM-7B, Calibration Variant=Base2026.050.48870.48820.564—
BaseGenerator Model=Llama-3.1-8B, Scorer Model=Qwen2.5-Math-PRM-7B, Calibration Variant=Base2026.050.50550.50350.6166—
ReasonEval-7BBackbone=Qwen-2.5-7B, Protocol=OT2026.050.54690.54230.66840.2698
ReasonEval-7BBackbone=Qwen-2.5-1.5B, Protocol=Base2026.050.550.5470.667—
Shepherd-7B (Qwen-2.5-1.5B)Backbone=Qwen-2.5-1.5B, Calibration=OT2026.050.58510.58410.71430.3175
BaseGenerator Model=Qwen-2.5-1.5B, Scorer Model=Qwen2.5-Math-PRM-7B, Calibration Variant=Base2026.050.59580.59480.6836—
ReasonEval-7BBackbone=R1-Llama-8B, Protocol=Base2026.050.70160.70110.7654—
BaseGenerator Model=R1-Llama-8B, Scorer Model=Qwen2.5-Math-PRM-7B, Calibration Variant=Base2026.050.70390.70310.7652—
ReasonEval-7BBackbone=R1-Qwen-7B, Protocol=Base2026.050.71110.71060.767—
BaseGenerator Model=R1-Qwen-7B, Scorer Model=Qwen2.5-Math-PRM-7B, Calibration Variant=Base2026.050.71580.71530.7703—