List your company
ResearchFormerly WizWand

Mathematical Reasoning on GSM8k (Accuracy)

100AccuracyPhi-4 pass@N (Upper Bound)
82.8487.29591.7596.205Mar 7, 2024Jul 18, 2024Nov 29, 2024Apr 12, 2025Aug 23, 2025Jan 4, 2026May 18, 2026
Updated 5mo ago

Evaluation results

MethodLinks
Phi-4 pass@N (Upper Bound)# Sample=-2025.11100
Majority Voting# Sample=-2025.11100
H4-Qwen2.5-PRM-1.5B-0.2# Sample=369K2025.11100
Universal-PRM-Qwen2.5-Math-7B# Sample=690K2025.11100
Qwen2.5-Math-7B-PRM800k# Sample=263K2025.11100
Qwen2.5-Math-7B-PRM# Sample=860K2025.11100
ReProbe, Attn+Logit, Qwen3-8B-anno# Sample=32K2025.11100
Skywork-PRM-1.5B# Sample=Unk2025.1199.5
RLHFlow-PRM-Deepseek-Data# Sample=253K2025.1198.9
Math-Shepherd-PRM-7B# Sample=440K2025.1198.4
RLHFlow-PRM-Mistral-Data# Sample=273K2025.1198.4
Phi-4 pass@1 (Lower Bound)# Sample=-2025.1197.9
SQ-formatModel=DeepSeek-R1, Setting=W(SQ5)A8, Sparsity=0.875, Evaluation Protocol=Generative2025.1296.21
DeepSeek-R1 (BF16)Model=DeepSeek-R1, Setting=W16A16, Sparsity=0, Evaluation Protocol=Generative2025.1295.83
ZEDABase Model=Qwen3-30B-A3B2026.0595.5
Qwen3-30B-A3BBase Model=Qwen3-30B-A3B2026.0595.4
NETSFT→OPDBase Model=Qwen3-30B-A3B2026.0595.4
Dynamic SkippingBase Model=Qwen3-30B-A3B2026.0595.2
GLM-4.7-FlashBase Model=GLM-4.7-Flash2026.0595.2
ZEDASFTBase Model=GLM-4.7-Flash2026.0595.2
ZEDASFTBase Model=Qwen3-30B-A3B2026.0594.8
NETSFTBase Model=Qwen3-30B-A3B2026.0594.7
ZEDABase Model=GLM-4.7-Flash2026.0594.4
NETSFT→OPDBase Model=GLM-4.7-Flash2026.0594.2
NETSFTBase Model=GLM-4.7-Flash2026.0594.1
AdaMoEBase Model=GLM-4.7-Flash2026.0593.9
Dynamic SkippingBase Model=GLM-4.7-Flash2026.0593.8
SRQ-driven steeringBackbone=Qwen3-4B-Thinking, Steering Mechanism=Conceptor2026.0493.3
SRQ-driven steeringBackbone=Qwen3-4B-Thinking, Steering Mechanism=PCA-CAA2026.0493.1
PPoT + Qwen2.5 CoderModel Size=7B, k (LLM samples)=5, m (PPoT samples)=52026.0493.02
DS-R1-Distill-Qwen-7B-ScaleQuestModel=DS-R1-Distill-Qwen-7B-ScaleQuest, Data=ScaleQuest2024.1093
SRQ-driven steeringBackbone=Qwen3-4B-Thinking, Steering Mechanism=CAA2026.0493
AUTOIF (LLaMA-3-70B w/ Online DPO)Backbone Model=LLaMA3-70B-Instruct, Supervision Model=LLaMA3-70B, Alignment Strategy=Self-Alignment, Training Method=Online DPO2024.0692.7
LLaMA3-70B-InstructBackbone Model=LLaMA3-70B-Instruct2024.0692.6
ConceptorBackbone=Qwen3-4B-Thinking2026.0492.6
PCA-CAABackbone=Qwen3-4B-Thinking2026.0492.4
AdaMoEBase Model=Qwen3-30B-A3B2026.0592.4
CAABackbone=Qwen3-4B-Thinking2026.0492.2
GPT-42024.0392
Base LLMBackbone=Qwen3-4B-Thinking2026.0491.9
VanillaTraining Strategy=Vanilla, Backbone=Qwen-2.5-32B-Instruct2026.0191.66
DS-R1-Distill-Qwen-7BModel=DS-R1-Distill-Qwen-7B2024.1091.5
Ans. AugTraining Strategy=Answer-Based Augmentation, Backbone=Qwen-2.5-32B-Instruct2026.0191.5
Qwen2.5 CoderModel Size=7B, k (LLM samples)=5, m (PPoT samples)=02026.0491.2
SATTraining Strategy=Structure-Aware Training, Backbone=Qwen-2.5-32B-Instruct2026.0191
BF16Model=Qwen-3 30B-A3B, Setting=W16A162025.1290.45
ES + TT-MVModel=OLMo3-7B-Inst2026.0390.2
SRQ-driven steeringBackbone=R1-Distill-Qwen-7B, Steering Mechanism=PCA-CAA2026.0489.9
SQ-formatModel=Qwen-3 30B-A3B, Setting=W4A(SQ5)2025.1289.84
SRQ-driven steeringBackbone=R1-Distill-Qwen-7B, Steering Mechanism=Conceptor2026.0489.6
RandOptModel=OLMo3-7B-Inst2026.0389.5
SRQ-driven steeringBackbone=R1-Distill-Qwen-7B, Steering Mechanism=CAA2026.0489.5
ESModel=OLMo3-7B2026.0389.1
PCA-CAABackbone=R1-Distill-Qwen-7B2026.0489.1
PPoT + Qwen2.5 CoderModel Size=3B, k (LLM samples)=5, m (PPoT samples)=52026.0489.08
Know. AugTraining Strategy=Knowledge-Based Augmentation, Backbone=Qwen-2.5-32B-Instruct2026.0188.8
SQ-formatModel=Qwen-3 30B-A3B, Setting=W4A(SQ6)2025.1288.78
ConceptorBackbone=R1-Distill-Qwen-7B2026.0488.6
Qwen3-8BCategory=Naive LLM2026.0188.42
PPOModel=OLMo3-7B-Inst2026.0388.4
CAABackbone=R1-Distill-Qwen-7B2026.0488.4
SRQ-driven steeringBackbone=R1-Distill-Llama-8B, Steering Mechanism=PCA-CAA2026.0488.4
AUTOIF (Qwen2-72B-Instruct w/ Online DPO)Backbone Model=Qwen2-72B-Instruct, Supervision Model=Qwen2-72B, Alignment Strategy=Self-Alignment, Training Method=Online DPO2024.0688.2
SRQ-driven steeringBackbone=R1-Distill-Llama-8B, Steering Mechanism=Conceptor2026.0488.2
SQ-formatModel=Qwen-3 30B-A3B, Setting=W(SQ6)A42025.1288.17
SRQ-driven steeringBackbone=R1-Distill-Llama-8B, Steering Mechanism=CAA2026.0488.1
Qwen2-72B-InstructBackbone Model=Qwen2-72B-Instruct2024.0687.9
ES + TT-MVModel=Qwen2.5-3B-Inst2026.0387.9
Base LLMBackbone=R1-Distill-Qwen-7B2026.0487.8
SDAR-4B-ChatModel=SDAR-4B-Chat, Decoding Strategy=FourierSampler2026.0187.64
ES + TT-MVModel=Llama3.1-8B-Inst2026.0387.5
PCA-CAABackbone=R1-Distill-Llama-8B2026.0487.5
SDAR-4B-ChatModel=SDAR-4B-Chat, Decoding Strategy=RWS2026.0187.41
ESModel=OLMo3-7B-Inst2026.0387.2
VIST2-8BCategory=Ours2026.0187.15
RandOptModel=Qwen2.5-3B-Inst2026.0387.1
ConceptorBackbone=R1-Distill-Llama-8B2026.0487.1
GRPOModel=OLMo3-7B-Inst2026.0387
RandOptModel=Llama3.1-8B-Inst2026.0386.7
CAABackbone=R1-Distill-Llama-8B2026.0486.7
SDAR-4B-ChatModel=SDAR-4B-Chat, Decoding Strategy=Vanilla2026.0186.58
ES + TT-MVModel=OLMo3-7B2026.0386.4
Qwen2.5 CoderModel Size=3B, k (LLM samples)=5, m (PPoT samples)=02026.0486.2
Qwen3-4BCategory=Naive LLM2026.0185.97
ESModel=Qwen2.5-3B-Inst2026.0385.8
Base LLMBackbone=R1-Distill-Llama-8B2026.0485.8
FP16Format=FP162025.0985.1
Best-of-N‡Model=OLMo3-7B-Inst2026.0385
RandOptModel=OLMo3-7B2026.0385
RTNFormat=INT82025.0984.8
PPoT + Qwen2.5 CoderModel Size=7B, k (LLM samples)=1, m (PPoT samples)=52026.0484.76
RTNFormat=FP82025.0984.7
GRPO (GT)Model=Qwen2.5-3B, Data=MATH2026.0184.61
GPTQFormat=FP82025.0984.5
GPTQFormat=INT82025.0984.4
PRISMModel=Qwen2.5-7B, Data=DAPO-17k2026.0184.31
VIST2-4BCategory=Ours2026.0184.31
Qwen2.5-14BPrecision=FP162026.0484
Bit-by-BitBackbone=Qwen2.5-14B, Precision=w2a162026.0484
Best-of-N‡Model=Llama3.1-8B-Inst2026.0383.5