List your company
ResearchFormerly WizWand

Mathematical Reasoning on ASDiv (Pass@1)

95.3Pass@1Baseline
67.01274.35681.789.044Mar 4, 2024Jun 9, 2024Sep 14, 2024Dec 21, 2024Mar 28, 2025Jul 3, 2025Oct 9, 2025
Updated 5mo ago

Evaluation results

MethodLinks
BaselineBase Model=Qwen2.5-Math-7B-Instruct, Evaluation Protocol=Zero-shot2025.1095.3
SFTBase Model=Qwen2.5-Math-7B-Instruct, Evaluation Protocol=Zero-shot2025.1095.2
LightRBase Model=Qwen2.5-Math-7B-Instruct, Evaluation Protocol=Zero-shot2025.1095.2
Qwen2.5-7B + GRPO w/ VERL.Base Model=Qwen2.5-7B, RL Framework=GRPO, VERL Integration=true2025.0995
Qwen2.5-7B + PPOBase Model=Qwen2.5-7B, RL Framework=PPO, VERL Integration=false2025.0994.9
Qwen2.5-7B + GRPOBase Model=Qwen2.5-7B, RL Framework=GRPO, VERL Integration=false2025.0994.8
Qwen2.5-7B + PPO w/ VERL.Base Model=Qwen2.5-7B, RL Framework=PPO, VERL Integration=true2025.0994.8
BaselineBase Model=Qwen2.5-Math-1.5B-Instruct, Evaluation Protocol=Zero-shot2025.1094.7
SFTBase Model=Qwen2.5-Math-1.5B-Instruct, Evaluation Protocol=Zero-shot2025.1094.7
LightRBase Model=Qwen2.5-Math-1.5B-Instruct, Evaluation Protocol=Zero-shot2025.1094.1
Qwen2.5-7BBase Model=Qwen2.5-7B, RL Framework=None, VERL Integration=false2025.0991.4
GPT-4 (0613)Zero-shot without demonstrations=false2024.0391.3
LEMMA (w/ MetaMath)Backbone=DeepSeekMath-7B, # Samples=403.59k2025.0390.4
LEMMABackbone=DeepSeekMath-7B, # Samples=88.90k2025.0389.8
Llama-3.2-3B-Instruct + GRPO w/ VERL.Base Model=Llama-3.2-3B-Instruct, RL Framework=GRPO, VERL Integration=true2025.0989.3
KPMath-PlusBase=Qwen1.5, Size=72B, Zero-shot without demonstrations=true2024.0389.2
KPMath-PlusBase=Llama-2, Size=70B, Zero-shot without demonstrations=true2024.0389
KPMath-PlusBase=DSMath, Size=7B, Zero-shot without demonstrations=true2024.0388.9
Llama-3.2-3B-Instruct + GRPOBase Model=Llama-3.2-3B-Instruct, RL Framework=GRPO, VERL Integration=false2025.0988.8
GPTAugBackbone=DeepSeekMath-7B, # Samples=88.62k2025.0388.7
Llama-3.2-3B-Instruct + PPO w/ VERL.Base Model=Llama-3.2-3B-Instruct, RL Framework=PPO, VERL Integration=true2025.0988.7
Llama-3.2-3B-Instruct + PPOBase Model=Llama-3.2-3B-Instruct, RL Framework=PPO, VERL Integration=false2025.0987.9
MetaMathBackbone=DeepSeekMath-7B, # Samples=394.99k2025.0387.8
KPMath-PlusBase=Llemma, Size=34B, Zero-shot without demonstrations=true2024.0387.5
LightRBase Model=DeepSeek-R1-Distill-Qwen-1.5B, Evaluation Protocol=Zero-shot2025.1087.5
SFTBase Model=DeepSeek-R1-Distill-Qwen-1.5B, Evaluation Protocol=Zero-shot2025.1087.4
ChatGPTZero-shot without demonstrations=false2024.0387.3
LEMMA (w/ MetaMath)Backbone=LLaMA3-8B, # Samples=403.59k2025.0387.1
KPMath-PlusBase=Mistral, Size=7B, Zero-shot without demonstrations=true2024.0386.7
RFTBackbone=DeepSeekMath-7B, # Samples=86.52k2025.0385.2
BaselineBase Model=DeepSeek-R1-Distill-Qwen-1.5B, Evaluation Protocol=Zero-shot2025.1084.9
LEMMABackbone=LLaMA3-8B, # Samples=88.90k2025.0384.2
KPMath-PlusBase=Llama-2, Size=13B, Zero-shot without demonstrations=true2024.0383.2
RefAug-90kBackbone=DeepSeekMath-7B, # Samples=89.92k2025.0382.4
RefAugBackbone=LLaMA3-8B, # Samples=29.94k2025.0382.3
ISCBackbone=DeepSeekMath-7B, # Samples=86.78k2025.0382.2
RefAug-90kBackbone=LLaMA3-8B, # Samples=89.92k2025.0382.1
MetaMathBackbone=LLaMA3-8B, # Samples=394.99k2025.0381.9
GPTAugBackbone=LLaMA3-8B, # Samples=88.62k2025.0381.2
RefAugBackbone=DeepSeekMath-7B, # Samples=29.94k2025.0381.2
ISCBackbone=LLaMA3-8B, # Samples=86.78k2025.0381.1
SFTBackbone=DeepSeekMath-7B, # Samples=14.97k2025.0380.9
LightRBase Model=Qwen2.5-Math-7B, Evaluation Protocol=Zero-shot2025.1080.6
LightRBase Model=Qwen2.5-Math-1.5B, Evaluation Protocol=Zero-shot2025.1079.8
SFTBase Model=Qwen2.5-Math-7B, Evaluation Protocol=Zero-shot2025.1076.6
RFTBackbone=LLaMA3-8B, # Samples=86.52k2025.0374.8
Llama-3.2-3B-InstructBase Model=Llama-3.2-3B-Instruct, RL Framework=None, VERL Integration=false2025.0974.6
BaselineBase Model=Qwen2.5-Math-7B, Evaluation Protocol=Zero-shot2025.1072.7
SFTBackbone=LLaMA3-8B, # Samples=14.97k2025.0372.1
SFTBase Model=Qwen2.5-Math-1.5B, Evaluation Protocol=Zero-shot2025.1070.2
BaselineBase Model=Qwen2.5-Math-1.5B, Evaluation Protocol=Zero-shot2025.1068.1