List your company
ResearchFormerly WizWand

Mathematical Reasoning on GSM8K v1 (test)

95.1AccuracyPass@k
55.47665.76376.0586.337Jul 15, 2024Nov 6, 2024Feb 28, 2025Jun 23, 2025Oct 15, 2025Feb 6, 2026Jun 1, 2026
Updated 5mo ago

Evaluation results

MethodLinks
Pass@kModel Pair=Qwen2.5-1.5B → Qwen2.5-32B, Sample Size (k)=322026.0695.1——
Large greedyModel Pair=Llama-3.2-1B → Llama-3.1-70B, Sample Size (k)=82026.0695.1——
Large greedyModel Pair=Llama-3.2-1B → Llama-3.1-70B, Sample Size (k)=162026.0695.1——
Large greedyModel Pair=Llama-3.2-1B → Llama-3.1-70B, Sample Size (k)=322026.0695.1——
NEMOTRON-NANO-12B-V2 (RLP + Post)Pre-training objective=RLP, Post-training pipeline=SFT + RLVR, Evaluation protocol=Pass@1 (average of 8 runs)2025.0994.9——
Large greedyModel Pair=Qwen2.5-1.5B → Qwen2.5-32B, Sample Size (k)=82026.0694.8——
Large greedyModel Pair=Qwen2.5-1.5B → Qwen2.5-32B, Sample Size (k)=162026.0694.8——
Large greedyModel Pair=Qwen2.5-1.5B → Qwen2.5-32B, Sample Size (k)=322026.0694.8——
NEMOTRON-NANO-12B-V2 (Base + Post)Pre-training objective=Next-token prediction, Post-training pipeline=SFT + RLVR, Evaluation protocol=Pass@1 (average of 8 runs)2025.0994.22——
Vanilla SPDraft Model=Llama-3.1-8B-Instruct, Target Model=Llama-3.1-70B-Instruct2026.019324.19—
AutoJudgeDraft Model=Llama-3.1-8B-Instruct, Target Model=Llama-3.1-70B-Instruct, Threshold=0.042026.0192.7727.341.13
Pass@kModel Pair=Qwen2.5-1.5B → Qwen2.5-32B, Sample Size (k)=162026.0692.7——
KLDraft Model=Llama-3.1-8B-Instruct, Target Model=Llama-3.1-70B-Instruct, Threshold=0.202026.0192.6927.211.12
AutoJudgeDraft Model=Llama-3.1-8B-Instruct, Target Model=Llama-3.1-70B-Instruct, Threshold=0.072026.0192.5429.551.22
CGSModel Pair=Qwen2.5-1.5B → Qwen2.5-32B, Sample Size (k)=32, Chunk Length (L)=202026.0692.5——
PRM guided searchModel Pair=Qwen2.5-1.5B → Qwen2.5-32B, Sample Size (k)=322026.0692.3——
AutoJudgeDraft Model=Llama-3.1-8B-Instruct, Target Model=Llama-3.1-70B-Instruct, Threshold=0.092026.0192.0930.981.28
KLDraft Model=Llama-3.1-8B-Instruct, Target Model=Llama-3.1-70B-Instruct, Threshold=0.402026.0191.9430.11.24
KLDraft Model=Llama-3.1-8B-Instruct, Target Model=Llama-3.1-70B-Instruct, Threshold=0.502026.0191.8831.491.3
KLDraft Model=Llama-3.1-8B-Instruct, Target Model=Llama-3.1-70B-Instruct, Threshold=0.902026.0191.2833.211.37
AutoJudgeDraft Model=Llama-3.1-8B-Instruct, Target Model=Llama-3.1-70B-Instruct, Threshold=0.142026.0191.0832.571.35
LGSModel Pair=Qwen2.5-1.5B → Qwen2.5-32B, Sample Size (k)=32, Chunk Length (L)=202026.0690.4——
KLDraft Model=Llama-3.1-8B-Instruct, Target Model=Llama-3.1-70B-Instruct, Threshold=1.402026.0190.2735.11.45
AutoJudgeDraft Model=Llama-3.1-8B-Instruct, Target Model=Llama-3.1-70B-Instruct, Threshold=0.222026.0189.9635.071.45
CGSModel Pair=Qwen2.5-1.5B → Qwen2.5-32B, Sample Size (k)=16, Chunk Length (L)=202026.0689.3——
PRM guided searchModel Pair=Qwen2.5-1.5B → Qwen2.5-32B, Sample Size (k)=162026.0689.2——
Pass@kModel Pair=Qwen2.5-1.5B → Qwen2.5-32B, Sample Size (k)=82026.0688.6——
LGSModel Pair=Qwen2.5-1.5B → Qwen2.5-32B, Sample Size (k)=16, Chunk Length (L)=202026.0687.6——
PROMPTCOT-Qwen-1.5BCoT setting=short-CoT, Base model=Qwen2.5-Math-1.5B2025.0387.1——
Pass@kModel Pair=Llama-3.2-1B → Llama-3.1-70B, Sample Size (k)=322026.0686.8——
PROMPTCOT-DS-1.5BCoT setting=long-CoT, Base model=DeepSeek-R1-Distill-Qwen-1.5B2025.0386.1——
PRM guided searchModel Pair=Qwen2.5-1.5B → Qwen2.5-32B, Sample Size (k)=82026.0686——
NEMOTRON-NANO-12B-V2 (RLP)Pre-training objective=RLP, Post-training pipeline=None, Evaluation protocol=Pass@1 (average of 8 runs)2025.0985.98——
Qwen2-7BInstruction-tuned=true2024.0785.7——
CGSModel Pair=Qwen2.5-1.5B → Qwen2.5-32B, Sample Size (k)=8, Chunk Length (L)=202026.0685.7——
LGSModel Pair=Qwen2.5-1.5B → Qwen2.5-32B, Sample Size (k)=8, Chunk Length (L)=202026.0685.4——
NuminaMath-1.5BCoT setting=short-CoT, Base model=Qwen2.5-Math-1.5B2025.0385.1——
DeepSeek-R1-Distill-Qwen-1.5BCoT setting=long-CoT, reproduced_using_our_prompt=true2025.0385.1——
Openmathinstruct-1.5BCoT setting=short-CoT, Base model=Qwen2.5-Math-1.5B2025.0384.9——
Yi-1.5-9BInstruction-tuned=true2024.0784.8——
Qwen2.5-Math-1.5B-InstructCoT setting=short-CoT, Base model=N/A2025.0384.8——
Evol-Instruct-1.5BCoT setting=short-CoT, Base model=Qwen2.5-Math-1.5B2025.0384.6——
TABOMSFT Training Domain=Math SFT (MixChain-Z-PRM12K)2026.0584.31——
CGSModel Pair=Llama-3.2-1B → Llama-3.1-70B, Sample Size (k)=32, Chunk Length (L)=202026.0683.9——
KPDDS-1.5BCoT setting=short-CoT, Base model=Qwen2.5-Math-1.5B2025.0383.4——
KLDraft Model=Llama-3.2-1B-Instruct, Target Model=Llama-3.1-8B-Instruct, Threshold=0.102026.0182.7739.581.03
AutoJudgeDraft Model=Llama-3.2-1B-Instruct, Target Model=Llama-3.1-8B-Instruct, Threshold=0.012026.0182.539.111.02
Vanilla SPDraft Model=Llama-3.2-1B-Instruct, Target Model=Llama-3.1-8B-Instruct2026.0182.538.4—
SFT-SDSFT Training Domain=Math SFT (MixChain-Z-PRM12K)2026.0581.95——
SFT-SDSFT Training Domain=Code SFT (Ling-Coder-SFT)2026.0581.81——
TABOMSFT Training Domain=Code SFT (Ling-Coder-SFT)2026.0581.73——
No-SFTSFT Training Domain=None (Base Model)2026.0581.41——
LGSModel Pair=Llama-3.2-1B → Llama-3.1-70B, Sample Size (k)=32, Chunk Length (L)=202026.0681.2——
KLDraft Model=Llama-3.2-1B-Instruct, Target Model=Llama-3.1-8B-Instruct, Threshold=0.402026.0181.0748.351.26
Borda count (large)Model Pair=Qwen2.5-1.5B → Qwen2.5-32B, Sample Size (k)=322026.0681——
KLDraft Model=Llama-3.2-1B-Instruct, Target Model=Llama-3.1-8B-Instruct, Threshold=0.302026.0180.9645.351.18
Best-of-NModel Pair=Qwen2.5-1.5B → Qwen2.5-32B, Sample Size (k)=322026.0680.7——
Best-of-NModel Pair=Qwen2.5-1.5B → Qwen2.5-32B, Sample Size (k)=162026.0680.5——
AutoJudgeDraft Model=Llama-3.2-1B-Instruct, Target Model=Llama-3.1-8B-Instruct, Threshold=0.072026.0180.2945.061.17
SFT-GTSFT Training Domain=Math SFT (MixChain-Z-PRM12K)2026.0580.12——
Borda count (small)Model Pair=Qwen2.5-1.5B → Qwen2.5-32B, Sample Size (k)=322026.0680.1——
Majority@kModel Pair=Qwen2.5-1.5B → Qwen2.5-32B, Sample Size (k)=322026.0679.7——
Llama-3-8BInstruction-tuned=true2024.0779.6——
GLM-4-9BInstruction-tuned=true2024.0779.6——
Best-of-NModel Pair=Qwen2.5-1.5B → Qwen2.5-32B, Sample Size (k)=82026.0679.6——
KLDraft Model=Llama-3.2-1B-Instruct, Target Model=Llama-3.1-8B-Instruct, Threshold=0.502026.0179.4650.961.33
AutoJudgeDraft Model=Llama-3.2-1B-Instruct, Target Model=Llama-3.1-8B-Instruct, Threshold=0.092026.0179.3947.121.23
CGSModel Pair=Llama-3.2-1B → Llama-3.1-70B, Sample Size (k)=16, Chunk Length (L)=202026.0679.3——
Pass@kModel Pair=Llama-3.2-1B → Llama-3.1-70B, Sample Size (k)=162026.0679.1——
Borda count (large)Model Pair=Qwen2.5-1.5B → Qwen2.5-32B, Sample Size (k)=162026.0678.7——
AutoJudgeDraft Model=Llama-3.2-1B-Instruct, Target Model=Llama-3.1-8B-Instruct, Threshold=0.142026.0178.448.871.27
Borda count (small)Model Pair=Qwen2.5-1.5B → Qwen2.5-32B, Sample Size (k)=162026.0678——
Majority@kModel Pair=Qwen2.5-1.5B → Qwen2.5-32B, Sample Size (k)=162026.0677.7——
LGSModel Pair=Llama-3.2-1B → Llama-3.1-70B, Sample Size (k)=16, Chunk Length (L)=202026.0677.3——
PRM guided searchModel Pair=Llama-3.2-1B → Llama-3.1-70B, Sample Size (k)=322026.0677.3——
KLDraft Model=Llama-3.2-1B-Instruct, Target Model=Llama-3.1-8B-Instruct, Threshold=0.902026.0177.1361.451.6
AutoJudgeDraft Model=Llama-3.2-1B-Instruct, Target Model=Llama-3.1-8B-Instruct, Threshold=0.222026.0175.5953.71.4
Borda count (large)Model Pair=Qwen2.5-1.5B → Qwen2.5-32B, Sample Size (k)=82026.0675.2——
Borda count (small)Model Pair=Qwen2.5-1.5B → Qwen2.5-32B, Sample Size (k)=82026.0675——
Self-Certainty (large)Model Pair=Qwen2.5-1.5B → Qwen2.5-32B, Sample Size (k)=322026.0674.8——
Majority@kModel Pair=Qwen2.5-1.5B → Qwen2.5-32B, Sample Size (k)=82026.0674——
Self-Certainty (large)Model Pair=Qwen2.5-1.5B → Qwen2.5-32B, Sample Size (k)=162026.0673.4——
NEMOTRON-NANO-12B-V2 (Base)Pre-training objective=Next-token prediction, Post-training pipeline=None, Evaluation protocol=Pass@1 (average of 8 runs)2025.0972.31——
PRM guided searchModel Pair=Llama-3.2-1B → Llama-3.1-70B, Sample Size (k)=162026.0671.8——
CGSModel Pair=Llama-3.2-1B → Llama-3.1-70B, Sample Size (k)=8, Chunk Length (L)=202026.0671.3——
Pass@kModel Pair=Llama-3.2-1B → Llama-3.1-70B, Sample Size (k)=82026.0670.5——
LGSModel Pair=Llama-3.2-1B → Llama-3.1-70B, Sample Size (k)=8, Chunk Length (L)=202026.0669.8——
Self-Certainty (large)Model Pair=Qwen2.5-1.5B → Qwen2.5-32B, Sample Size (k)=82026.0669.7——
Self-Certainty (small)Model Pair=Qwen2.5-1.5B → Qwen2.5-32B, Sample Size (k)=162026.0667.9——
Self-Certainty (small)Model Pair=Qwen2.5-1.5B → Qwen2.5-32B, Sample Size (k)=322026.0667.5——
PRM guided searchModel Pair=Llama-3.2-1B → Llama-3.1-70B, Sample Size (k)=82026.0665.4——
Self-Certainty (small)Model Pair=Qwen2.5-1.5B → Qwen2.5-32B, Sample Size (k)=82026.0665.1——
Self-Certainty (large)Model Pair=Llama-3.2-1B → Llama-3.1-70B, Sample Size (k)=322026.0662.7——
Qwen1.5-7BInstruction-tuned=true2024.0760.3——
Best-of-NModel Pair=Llama-3.2-1B → Llama-3.1-70B, Sample Size (k)=322026.0660.2——
Self-Certainty (large)Model Pair=Llama-3.2-1B → Llama-3.1-70B, Sample Size (k)=162026.0659.2——
Borda count (large)Model Pair=Llama-3.2-1B → Llama-3.1-70B, Sample Size (k)=322026.0659.1——
Best-of-NModel Pair=Llama-3.2-1B → Llama-3.1-70B, Sample Size (k)=162026.0658.5——
Borda count (small)Model Pair=Llama-3.2-1B → Llama-3.1-70B, Sample Size (k)=322026.0657.2——
Borda count (large)Model Pair=Llama-3.2-1B → Llama-3.1-70B, Sample Size (k)=162026.0657——