Mathematical Problem Solving on AIME 2024 (Score)
100AccuracySelf Reflection
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| Self ReflectionModel=Gemini 3 Pro2025.09 | 100 | — | — | |
| CoTModel=Gemini 3 Pro2025.09 | 100 | — | — | |
| WaitModel=Gemini 3 Pro2025.09 | 100 | — | — | |
| HEARTModel=Gemini 3 Pro2025.09 | 100 | — | — | |
| VanillaModel=Gemini 3 Pro2025.09 | 97.5 | — | — | |
| HEARTModel=Gemini 2.5 Pro2025.09 | 95.83 | — | — | |
| Self ReflectionModel=Gemini 3 Flash2025.09 | 95.83 | — | — | |
| CoTModel=Gemini 2.5 Flash2025.09 | 95 | — | — | |
| HEARTModel=Gemini 2.5 Flash2025.09 | 95 | — | — | |
| Self ReflectionModel=Gemini 2.5 Flash2025.09 | 94.17 | — | — | |
| Self ReflectionModel=Gemini 2.5 Pro2025.09 | 94.17 | — | — | |
| Self ReflectionModel=Deepseek-Chat2025.09 | 93.33 | — | — | |
| WaitModel=Deepseek-Chat2025.09 | 93.33 | — | — | |
| HEARTModel=Deepseek-Chat2025.09 | 93.33 | — | — | |
| CoTModel=Deepseek-Chat2025.09 | 92.5 | — | — | |
| WaitModel=Gemini 2.5 Flash2025.09 | 91.67 | — | — | |
| CoTModel=Gemini 2.5 Pro2025.09 | 91.67 | — | — | |
| CoTModel=Gemini 3 Flash2025.09 | 91.67 | — | — | |
| HEARTModel=Gemini 3 Flash2025.09 | 91.67 | — | — | |
| Self ReflectionModel=GPT-5-nano2025.09 | 91.67 | — | — | |
| HEARTModel=GPT-5-nano2025.09 | 91.67 | — | — | |
| CoTModel=Deepseek-Reasoner2025.09 | 91.67 | — | — | |
| HEARTModel=Gemini-2.5-Flash (Thinking Budget set to 0)2025.09 | 91.67 | — | — | |
| Self ReflectionModel=Deepseek-Reasoner2025.09 | 90.83 | — | — | |
| WaitModel=Deepseek-Reasoner2025.09 | 90 | — | — | |
| WaitModel=Gemini-2.5-Flash (Thinking Budget set to 0)2025.09 | 90 | — | — | |
| HEARTModel=Deepseek-Reasoner2025.09 | 88.33 | — | — | |
| Self ReflectionModel=Gemini-2.5-Flash (Thinking Budget set to 0)2025.09 | 88.33 | — | — | |
| CoTModel=GPT-5-nano2025.09 | 87.59 | — | — | |
| WaitModel=Gemini 3 Flash2025.09 | 87.5 | — | — | |
| WaitModel=GPT-5-nano2025.09 | 87.5 | — | — | |
| WaitModel=Gemini 2.5 Pro2025.09 | 86.67 | — | — | |
| Majority VoteModel Pool=Gemini-2.5-flash2026.04 | 86.67 | — | — | |
| Self RefineModel Pool=Gemini-2.5-flash2026.04 | 86.67 | — | — | |
| ParaManagerStrategy=Model Orchestration2026.04 | 86.67 | — | — | |
| Majority VoteModel Pool=Qwen3-235B-A22B-Instruct-25072026.04 | 85.83 | — | — | |
| HEARTModel=Claude 4 Sonnet2025.09 | 85 | — | — | |
| CoTModel=Gemini-2.5-Flash (Thinking Budget set to 0)2025.09 | 84.17 | — | — | |
| Self RefineModel Pool=Qwen3-235B-A22B-Instruct-25072026.04 | 83.33 | — | — | |
| Majority VoteModel Pool=GPT-OSS-120B2026.04 | 83.33 | — | — | |
| Self RefineModel Pool=GPT-OSS-120B2026.04 | 83.33 | — | — | |
| ToolOrchestraStrategy=Model Orchestration2026.04 | 83.33 | — | — | |
| BaseModel Pool=Qwen3-235B-A22B-Instruct-25072026.04 | 80 | — | — | |
| PuppeteerStrategy=Model Orchestration2026.04 | 80 | — | — | |
| BaseModel Pool=Gemini-2.5-flash2026.04 | 78.75 | — | — | |
| HEARTModel=Claude-4-Sonnet (Thinking Off)2025.09 | 78.33 | — | — | |
| VanillaModel=Gemini 3 Flash2025.09 | 76.67 | — | — | |
| VanillaModel=Gemini-2.5-Flash (Thinking Budget set to 0)2025.09 | 76.67 | — | — | |
| TAGMemory setting=Our curated AIME memory2026.04 | 76.67 | — | 23 | |
| BaseModel Pool=GPT-OSS-120B2026.04 | 76.25 | — | — | |
| Self ReflectionModel=Claude-4-Sonnet (Thinking Off)2025.09 | 75 | — | — | |
| WaitModel=Claude-4-Sonnet (Thinking Off)2025.09 | 75 | — | — | |
| Official no-memory baselineMemory setting=Official no-memory2026.04 | 73.33 | — | 22 | |
| Self ReflectionModel=Claude 4 Sonnet2025.09 | 72.5 | — | — | |
| VanillaModel=GPT-5-nano2025.09 | 70.83 | — | — | |
| CoTModel=Claude-4-Sonnet (Thinking Off)2025.09 | 70.83 | — | — | |
| VanillaModel=Gemini 2.5 Flash2025.09 | 70 | — | — | |
| VanillaModel=Gemini 2.5 Pro2025.09 | 70 | — | — | |
| CoTModel=Claude 4 Sonnet2025.09 | 70 | — | — | |
| Official TFGRPO memoryMemory setting=retrained2026.04 | 70 | — | 21 | |
| WaitModel=Claude 4 Sonnet2025.09 | 69.17 | — | — | |
| SFTModel=DS-R1-Qwen-32B2026.04 | 67.5 | — | — | |
| ReToolModel=Qwen3-8B, SFT=✓2026.03 | 67 | — | — | |
| VanillaModel=Deepseek-Chat2025.09 | 66.67 | — | — | |
| ICRLModel=Qwen3-8B, SFT=✗2026.03 | 64.1 | — | — | |
| VanillaModel=Claude 4 Sonnet2025.09 | 63.33 | — | — | |
| Official TFGRPO memoryMemory setting=released2026.04 | 63.33 | — | 19 | |
| D-QRELOModel=DS-R1-Qwen-32B2026.04 | 62.5 | — | — | |
| SFTModel=DS-R1-Qwen-14B2026.04 | 61.67 | — | — | |
| D-QRELOModel=DS-R1-Qwen-14B2026.04 | 57.5 | — | — | |
| HEARTModel=Gemma3 12b Instruct2025.09 | 56.67 | — | — | |
| VanillaModel=Claude-4-Sonnet (Thinking Off)2025.09 | 55.83 | — | — | |
| Delta-CoMeModel=DS-R1-Qwen-32B2026.04 | 53.33 | — | — | |
| VanillaModel=Deepseek-Reasoner2025.09 | 50.83 | — | — | |
| Delta-CoMeModel=DS-R1-Qwen-14B2026.04 | 47 | — | — | |
| CoTModel=Gemma3 12b Instruct2025.09 | 45.83 | — | — | |
| WaitModel=Gemma3 12b Instruct2025.09 | 44.17 | — | — | |
| SFTModel=DS-R1-LLaMA-8B2026.04 | 44.17 | — | — | |
| Self ReflectionModel=Gemma3 12b Instruct2025.09 | 40.83 | — | — | |
| CoTModel=Gemma3 4b Instruct2025.09 | 35 | — | — | |
| HEARTModel=Gemma3 4b Instruct2025.09 | 32.5 | — | — | |
| WaitModel=Gemma3 4b Instruct2025.09 | 31.67 | — | — | |
| Self ReflectionModel=Gemma3 4b Instruct2025.09 | 30.83 | — | — | |
| VanillaModel=Gemma3 12b Instruct2025.09 | 25.83 | — | — | |
| D-QRELOModel=DS-R1-LLaMA-8B2026.04 | 23.83 | — | — | |
| MagnitudeModel=DS-R1-Qwen-32B2026.04 | 20.83 | — | — | |
| WandaModel=DS-R1-Qwen-32B2026.04 | 20.83 | — | — | |
| HAPOBackbone=LLaMA3.1-8B-Instruct2025.09 | 19.17 | — | — | |
| WandaModel=DS-R1-Qwen-14B2026.04 | 17.67 | — | — | |
| SVDModel=DS-R1-Qwen-32B2026.04 | 17.67 | — | — | |
| HAPOBackbone=LLaMA3.2-3B Instruct2025.09 | 17.5 | — | — | |
| MagnitudeModel=DS-R1-Qwen-14B2026.04 | 17.33 | — | — | |
| SVDModel=DS-R1-Qwen-14B2026.04 | 15.5 | — | — | |
| DAPO w/ Forking TokensBackbone=LLaMA3.1-8B-Instruct2025.09 | 14.5 | — | — | |
| VanillaModel=Gemma3 4b Instruct2025.09 | 14.17 | — | — | |
| Vanilla DAPOBackbone=LLaMA3.2-3B Instruct2025.09 | 13.93 | — | — | |
| Vanilla DAPOBackbone=LLaMA3.1-8B-Instruct2025.09 | 13.33 | — | — | |
| DAPO w/ Forking TokensBackbone=LLaMA3.2-3B Instruct2025.09 | 12.45 | — | — | |
| Vanilla GRPOBackbone=LLaMA3.2-3B Instruct2025.09 | 11.67 | — | — | |
| Vanilla GRPOBackbone=LLaMA3.1-8B-Instruct2025.09 | 10.83 | — | — |