Mathematical Reasoning on Minerva (Pass@1 Accuracy, Avg Length)
54.53Pass@1 AccuracyPass@k†
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Pass@k†Backbone=Phi-4-mini-Instruct, N=322026.05 | 54.53 | 17,119 | |
| Pass@k†Backbone=Qwen2.5-7B-Instruct, N=322026.05 | 52.33 | 19,765 | |
| ExpThinkBackbone=Qwen3-8B2026.05 | 45.5 | 850.4 | |
| VanillaBackbone=Qwen3-8B2026.05 | 44.1 | 7,210 | |
| LaserBase Model=DeepSeek-R1-Distill-Qwen, Model Scale=7B2026.03 | 43.9 | 1,826 | |
| SPSBackbone=Qwen2.5-7B-Instruct, N=82026.05 | 43.26 | 9,387 | |
| LC-R1Base Model=DeepSeek-R1-Distill-Qwen, Model Scale=7B2026.03 | 42.8 | 1,298.8 | |
| TRiMSBackbone=Qwen3-4B2026.03 | 41.9 | 783 | |
| PB-SMCBackbone=Qwen2.5-7B-Instruct, N=82026.05 | 41.79 | 13,216 | |
| VanillaBackbone=Qwen3-4B2026.03 | 41.5 | 6,673.1 | |
| JETBackbone=DeepSeek-R1-Distill-Qwen-7B2026.05 | 40.4 | 3,333.9 | |
| SMCBackbone=Qwen2.5-7B-Instruct, N=322026.05 | 40.32 | 21,642 | |
| Beam SearchBackbone=Qwen2.5-7B-Instruct, N=322026.05 | 40.2 | 23,900 | |
| DVTSBackbone=Qwen2.5-7B-Instruct, N=322026.05 | 39.22 | 23,549 | |
| TRiMSBase Model=DeepSeek-R1-Distill-Qwen, Model Scale=7B2026.03 | 39 | 696.8 | |
| Self-ConsistencyBackbone=Qwen2.5-7B-Instruct, N=322026.05 | 38.85 | 19,765 | |
| ExpThinkBackbone=DeepSeek-R1-Distill-Qwen-7B2026.05 | 38.6 | 1,182.7 | |
| Best-of-NBackbone=Qwen2.5-7B-Instruct, N=322026.05 | 38.48 | 19,765 | |
| PB-SMCBackbone=Phi-4-mini-Instruct, N=82026.05 | 37.87 | 13,198 | |
| AutoThinkBackbone=DeepSeek-R1-Distill-Qwen-7B2026.05 | 37.8 | 2,617.4 | |
| JETBase Model=DeepSeek-R1-Distill-Qwen, Model Scale=7B2026.03 | 37.5 | 1,786.1 | |
| VanillaBackbone=DeepSeek-R1-Distill-Qwen-7B2026.05 | 37.5 | 5,017.3 | |
| LaserBackbone=DeepSeek-R1-Distill-Qwen-7B2026.05 | 37.1 | 2,050.9 | |
| MCTSBackbone=Qwen2.5-7B-Instruct, N=322026.05 | 36.27 | 25,208 | |
| SPSBackbone=Phi-4-mini-Instruct, N=82026.05 | 35.42 | 9,832 | |
| AutoThinkBase Model=DeepSeek-R1-Distill-Qwen, Model Scale=7B2026.03 | 35.3 | 2,512.2 | |
| AdaptThinkBase Model=DeepSeek-R1-Distill-Qwen, Model Scale=7B2026.03 | 34.9 | 2,362 | |
| Best-of-NBackbone=Phi-4-mini-Instruct, N=322026.05 | 34.8 | 17,119 | |
| Base ModelBackbone=Qwen2.5-7B-Instruct, N=12026.05 | 34.68 | 617 | |
| SMCBackbone=Phi-4-mini-Instruct, N=322026.05 | 34.44 | 24,907 | |
| VanillaBase Model=DeepSeek-R1-Distill-Qwen, Model Scale=7B2026.03 | 34.2 | 4,907.5 | |
| DVTSBackbone=Phi-4-mini-Instruct, N=322026.05 | 33.82 | 22,140 | |
| AdaptThinkBackbone=DeepSeek-R1-Distill-Qwen-7B2026.05 | 33.8 | 2,645.4 | |
| Self-ConsistencyBackbone=Phi-4-mini-Instruct, N=322026.05 | 33.7 | 17,119 | |
| Beam SearchBackbone=Phi-4-mini-Instruct, N=322026.05 | 33.46 | 23,768 | |
| LaserBase Model=DeepSeek-R1-Distill-Qwen, Model Scale=1.5B2026.03 | 33.1 | 3,218.3 | |
| MCTSBackbone=Phi-4-mini-Instruct, N=322026.05 | 32.75 | 23,108 | |
| LC-R1Backbone=DeepSeek-R1-Distill-Qwen-7B2026.05 | 32.4 | 1,734.4 | |
| AutoThinkBase Model=DeepSeek-R1-Distill-Qwen, Model Scale=1.5B2026.03 | 29 | 3,031.5 | |
| ExpThinkBackbone=DeepSeek-R1-Distill-Qwen-1.5B2026.05 | 29 | 1,761 | |
| LaserBackbone=DeepSeek-R1-Distill-Qwen-1.5B2026.05 | 28.7 | 3,185.5 | |
| TRiMSBase Model=DeepSeek-R1-Distill-Qwen, Model Scale=1.5B2026.03 | 28.3 | 958.3 | |
| VanillaBackbone=DeepSeek-R1-Distill-Qwen-1.5B2026.05 | 26.8 | 6,347.1 | |
| LC-R1Base Model=DeepSeek-R1-Distill-Qwen, Model Scale=1.5B2026.03 | 26.5 | 3,827.3 | |
| LC-R1Backbone=DeepSeek-R1-Distill-Qwen-1.5B2026.05 | 26.1 | 2,383.3 | |
| VanillaBase Model=DeepSeek-R1-Distill-Qwen, Model Scale=1.5B2026.03 | 25.7 | 6,284.1 | |
| AdaptThinkBase Model=DeepSeek-R1-Distill-Qwen, Model Scale=1.5B2026.03 | 25.5 | 1,876.8 | |
| JETBackbone=DeepSeek-R1-Distill-Qwen-1.5B2026.05 | 25.4 | 3,988.1 | |
| AutoThinkBackbone=DeepSeek-R1-Distill-Qwen-1.5B2026.05 | 24.3 | 3,201.2 | |
| JETBase Model=DeepSeek-R1-Distill-Qwen, Model Scale=1.5B2026.03 | 23.9 | 1,839.1 | |
| Base ModelBackbone=Phi-4-mini-Instruct, N=12026.05 | 23.9 | 535 | |
| AdaptThinkBackbone=DeepSeek-R1-Distill-Qwen-1.5B2026.05 | 22.1 | 1,968.5 |