Mathematical Problem Solving on MATH (test)
93.22AccuracyCOPRO-M
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| COPRO-MLMforward=Gemini 2.5 flash-lite, LMbackward=Gemini 2.5 flash, Optimization Granularity=Promptwise2025.05 | 93.22 | — | 0.45 | |
| COPRO-MLMforward=Gemini 2.5 flash-lite, LMbackward=Gemini 2.5 flash, Optimization Granularity=Blockwise2025.05 | 93.01 | — | 0.59 | |
| TextGrad-MLMforward=Gemini 2.5 flash-lite, LMbackward=Gemini 2.5 flash, Optimization Granularity=Promptwise2025.05 | 91.58 | — | 1.58 | |
| TextGrad-MLMforward=Gemini 2.5 flash-lite, LMbackward=Gemini 2.5 flash, Optimization Granularity=Blockwise2025.05 | 91.02 | — | 1.72 | |
| COPROLMforward=Gemini 2.5 flash-lite, LMbackward=Gemini 2.5 flash2025.05 | 90.76 | — | 0.56 | |
| AdalFlow-MLMforward=Gemini 2.5 flash-lite, LMbackward=Gemini 2.5 flash, Optimization Granularity=Promptwise2025.05 | 90.73 | — | 0.006 | |
| AdalFlowLMforward=Gemini 2.5 flash-lite, LMbackward=Gemini 2.5 flash2025.05 | 89.72 | — | 0.005 | |
| GEPALMforward=Gemini 2.5 flash-lite, LMbackward=Gemini 2.5 flash2025.05 | 88 | — | 0.005 | |
| TextGradLMforward=Gemini 2.5 flash-lite, LMbackward=Gemini 2.5 flash, Validation Revert Strategy=w/o2025.05 | 87.58 | — | 2.12 | |
| TextGradLMforward=Gemini 2.5 flash-lite, LMbackward=Gemini 2.5 flash, Validation Revert Strategy=w/2025.05 | 83.36 | — | 0.32 | |
| JF-HPOModel=Qwen-3 14B2026.06 | 71.83 | — | — | |
| BOHBModel=Qwen-2.5 7B2026.06 | 70.29 | — | — | |
| VeRL RecipeModel=Qwen-3 14B2026.06 | 70.21 | — | — | |
| JF-HPOModel=Qwen-2.5 7B2026.06 | 68.19 | — | — | |
| Random SearchModel=Qwen-2.5 7B2026.06 | 66.93 | — | — | |
| VeRL RecipeModel=Qwen-2.5 7B2026.06 | 63.21 | — | — | |
| Gemini-UltraModel Category=Proprietary, Tool Usage=No, Eval Method=PASS@12024.01 | 53.2 | — | — | |
| TORA-CODE-34BModel Category=~34B MODELS, FT-Dataset=TORA-CORPUS, Tool Usage=CODE, Eval Method=PASS@12024.01 | 50.8 | — | — | |
| Minerva-540BModel Category=Proprietary, FT-Dataset=ARXIV+WEB, Tool Usage=No, Eval Method=MAJ1@642024.01 | 50.3 | — | — | |
| JF-HPOModel=LLaMA-3.1 8B2026.06 | 48.64 | — | — | |
| BOHBModel=LLaMA-3.1 8B2026.06 | 48.62 | — | — | |
| DEEPSEEK-67B-METAMATHQA (Verification)Model Category=~70B MODELS, FT-Dataset=METAMATHQA, Tool Usage=MATH-SHEPHERD, Eval Method=MAJ+VERIFY1@2562024.01 | 48.1 | — | — | |
| LLEMMA-34B-METAMATHQA (Verification)Model Category=~34B MODELS, FT-Dataset=METAMATHQA, Tool Usage=MATH-SHEPHERD, Eval Method=MAJ+VERIFY1@2562024.01 | 47.3 | — | — | |
| DEEPSEEK-67B-METAMATHQA (Majority Voting)Model Category=~70B MODELS, FT-Dataset=METAMATHQA, Tool Usage=No, Eval Method=MAJ1@2562024.01 | 45.4 | — | — | |
| QWEN-72B-MMIQCModel Category=~70B MODELS, FT-Dataset=MMIQC, Tool Usage=No, Eval Method=PASS@12024.01 | 45 | — | — | |
| TORA-CODE-7BModel Category=~7B MODELS, FT-Dataset=TORA-CORPUS, Tool Usage=CODE, Eval Method=PASS@12024.01 | 44.6 | — | — | |
| GPT-4 (2023-0314)Model Category=Proprietary, Tool Usage=No, Eval Method=PASS@12024.01 | 42.5 | — | — | |
| QWEN-72B-METAMATHQAModel Category=~70B MODELS, FT-Dataset=METAMATHQA, Tool Usage=No, Eval Method=PASS@12024.01 | 41.7 | — | — | |
| DEEPSEEK-67B-MMIQCModel Category=~70B MODELS, FT-Dataset=MMIQC, Tool Usage=No, Eval Method=PASS@12024.01 | 41 | — | — | |
| LLEMMA-34B-MMIQCModel Category=~34B MODELS, FT-Dataset=MMIQC, Tool Usage=No, Eval Method=PASS@12024.01 | 38.6 | — | — | |
| DEEPSEEK-67B-METAMATHQAModel Category=~70B MODELS, FT-Dataset=METAMATHQA, Tool Usage=No, Eval Method=PASS@12024.01 | 36.8 | — | — | |
| MISTRAL-7B-MMIQCModel Category=~7B MODELS, FT-Dataset=MMIQC, Tool Usage=No, Eval Method=PASS@12024.01 | 36 | — | — | |
| MAMMOTH-CODER-7BModel Category=~7B MODELS, FT-Dataset=MATHINSTRUCT, Tool Usage=CODE, Eval Method=PASS@12024.01 | 35.2 | — | — | |
| QWEN-72BModel Category=~70B MODELS, Tool Usage=No, Eval Method=PASS@12024.01 | 35.2 | — | — | |
| LLEMMA-34B-METAMATHQAModel Category=~34B MODELS, FT-Dataset=METAMATHQA, Tool Usage=No, Eval Method=PASS@12024.01 | 34.8 | — | — | |
| Random SearchModel=LLaMA-3.1 8B2026.06 | 30.97 | — | — | |
| MISTRAL-7B-METAMATHQAModel Category=~7B MODELS, FT-Dataset=METAMATHQA, Tool Usage=No, Eval Method=PASS@12024.01 | 28.2 | — | — | |
| CODELLAMMA-34BModel Category=~34B MODELS, Tool Usage=CODE, Eval Method=PASS@12024.01 | 25 | — | — | |
| VeRL RecipeModel=LLaMA-3.1 8B2026.06 | 22.99 | — | — | |
| METAMATH-7BModel Category=~7B MODELS, FT-Dataset=METAMATHQA, Tool Usage=No, Eval Method=PASS@12024.01 | 19.8 | — | — | |
| DEEPSEEK-67BModel Category=~70B MODELS, Tool Usage=No, Eval Method=PASS@12024.01 | 18.7 | — | — | |
| LLEMMA-7BModel Category=~7B MODELS, FT-Dataset=PROOF-PILE-2, Tool Usage=No, Eval Method=PASS@12024.01 | 18 | — | — | |
| LLAMA-2-70BModel Category=~70B MODELS, Tool Usage=No, Eval Method=PASS@12024.01 | 13.5 | — | — | |
| QWEN-7BModel Category=~7B MODELS, Tool Usage=No, Eval Method=PASS@12024.01 | 11.6 | — | — | |
| LLAMA-2-7BModel Category=~7B MODELS, Tool Usage=No, Eval Method=PASS@12024.01 | 2.5 | — | — | |
| Baseline2026.05 | — | 60.7 | — | |
| CROWall (minutes)=422026.05 | — | 80 | — | |
| GEPAWall (minutes)=202026.05 | — | 74 | — | |
| LLEMMA 34BSize=34B, k (samples)=256, Temperature (T)=0.6, Nucleus sampling (p)=0.952023.10 | — | 43.1 | — | |
| LLEMMA 7BSize=7B, k (samples)=256, Temperature (T)=0.6, Nucleus sampling (p)=0.952023.10 | — | 33.5 | — | |
| MetaHarnessWall (minutes)=1012026.05 | — | 79.3 | — | |
| Minerva 540BSize=540B, k (samples)=256, Temperature (T)=0.6, Nucleus sampling (p)=0.952023.10 | — | 50.3 | — | |
| Minerva 62BSize=62B, k (samples)=256, Temperature (T)=0.6, Nucleus sampling (p)=0.952023.10 | — | 43.4 | — | |
| Minerva 8BSize=8B, k (samples)=256, Temperature (T)=0.6, Nucleus sampling (p)=0.952023.10 | — | 25.4 | — |