Mathematical Reasoning on GSM8K (test) (Accuracy)
99AccuracyNo Defense
Evaluation results
| Method | Links | |
|---|---|---|
| No DefenseModel=Claude-3.5-Sonnet2025.05 | 99 | |
| ICDModel=Claude-3.5-Sonnet2025.05 | 99 | |
| Goal PrioritizationModel=Claude-3.5-Sonnet2025.05 | 99 | |
| SAGEModel=Claude-3.5-Sonnet2025.05 | 99 | |
| No DefenseModel=GPT-4o2025.05 | 98 | |
| Self-ReminderModel=Claude-3.5-Sonnet2025.05 | 98 | |
| IAModel=Claude-3.5-Sonnet2025.05 | 98 | |
| IAModel=GPT-4o-mini2025.05 | 97 | |
| Self-ReminderModel=GPT-4o2025.05 | 97 | |
| Self-ExaminationModel=GPT-4o2025.05 | 97 | |
| Goal PrioritizationModel=GPT-4o2025.05 | 97 | |
| SAGEModel=GPT-4o2025.05 | 97 | |
| DensePruning Ratio=0%, Backbone=Qwen3-8B, prompt=8-shot2025.12 | 96.74 | |
| MGRSBackbone=GPT-4o-mini2025.11 | 96.5 | |
| AFlowOptimizer=Claude-3.5-Sonnet, Executor=Claude-3.7-Sonnet2026.01 | 96.5 | |
| FLExBase Model=Qwen-32B, Model Family=Qwen-2.5 Instruct2026.01 | 96.4 | |
| MaASOptimizer=Claude-3.5-Sonnet, Executor=Claude-3.7-Sonnet2026.01 | 96.4 | |
| IAModel=GPT-4o2025.05 | 96 | |
| CoT-SCOptimizer=Claude-3.5-Sonnet, Executor=Claude-3.7-Sonnet2026.01 | 96 | |
| ADASOptimizer=Claude-3.5-Sonnet, Executor=Claude-3.7-Sonnet2026.01 | 96 | |
| BayesFlowOptimizer=Claude-3.5-Sonnet, Executor=Claude-3.7-Sonnet2026.01 | 96 | |
| TSBH (Pass@5)Model=Qwen2.5-7B2026.04 | 96 | |
| FLExBase Model=Qwen-72B, Model Family=Qwen-2.5 Instruct2026.01 | 95.7 | |
| RESP w/o re-genPruning Ratio=20%, Backbone=Qwen3-8B, prompt=8-shot2025.12 | 95.53 | |
| RESPPruning Ratio=20%, Backbone=Qwen3-8B, prompt=8-shot2025.12 | 95.53 | |
| FLExBase Model=Gemma-27B, Model Family=Gemma-3 Instruct2026.01 | 95.5 | |
| SCBase Model=Qwen-14B, Model Family=Qwen-2.5 Instruct2026.01 | 95.5 | |
| CoTOptimizer=Claude-3.5-Sonnet, Executor=Claude-3.7-Sonnet2026.01 | 95.2 | |
| SCBase Model=Gemma-27B, Model Family=Gemma-3 Instruct2026.01 | 95.1 | |
| RAGBase Model=Qwen-72B, Model Family=Qwen-2.5 Instruct2026.01 | 95.1 | |
| No DefenseModel=GPT-4o-mini2025.05 | 95 | |
| Self-ReminderModel=GPT-4o-mini2025.05 | 95 | |
| ICDModel=GPT-4o-mini2025.05 | 95 | |
| SAGEModel=GPT-4o-mini2025.05 | 95 | |
| AoTBackbone=GPT-4o-mini2025.11 | 95 | |
| FLExBase Model=Qwen-14B, Model Family=Qwen-2.5 Instruct2026.01 | 94.8 | |
| RAGBase Model=Qwen-32B, Model Family=Qwen-2.5 Instruct2026.01 | 94.8 | |
| SRBase Model=Qwen-72B, Model Family=Qwen-2.5 Instruct2026.01 | 94.8 | |
| SRBase Model=Gemma-27B, Model Family=Gemma-3 Instruct2026.01 | 94.7 | |
| ICL_bioBackbone=GPT-4o-mini, Variant=ICL_bio2024.04 | 94.5 | |
| RAGBase Model=Gemma-27B, Model Family=Gemma-3 Instruct2026.01 | 94.5 | |
| Bidirectional Curriculum Generation (Round 4)Data Volume=5,8732026.03 | 94.47 | |
| RAGBase Model=Qwen-14B, Model Family=Qwen-2.5 Instruct2026.01 | 94.4 | |
| Bidirectional Curriculum Generation (Round 2)Data Volume=1,6332026.03 | 94.24 | |
| ICL_mathBackbone=GPT-4o-mini, Variant=ICL_math2024.04 | 94.2 | |
| CoTBase Model=Gemma-27B, Model Family=Gemma-3 Instruct2026.01 | 94.2 | |
| SRBase Model=Qwen-32B, Model Family=Qwen-2.5 Instruct2026.01 | 94.1 | |
| Bidirectional Curriculum Generation (Round 1)Data Volume=8502026.03 | 94.01 | |
| Goal PrioritizationModel=GPT-4o-mini2025.05 | 94 | |
| Self-ExaminationModel=Claude-3.5-Sonnet2025.05 | 94 | |
| SCBase Model=Gemma-12B, Model Family=Gemma-3 Instruct2026.01 | 94 | |
| FoTBackbone=GPT-4o-mini, n=82025.11 | 94 | |
| Bidirectional Curriculum Generation (Round 3)Data Volume=4,5942026.03 | 93.93 | |
| CoTBase Model=Qwen-72B, Model Family=Qwen-2.5 Instruct2026.01 | 93.9 | |
| DenoiseFlowBackbone LLM=GPT-4o-mini, System Category=Autonomous Multi-agent2026.02 | 93.9 | |
| MASPOBBackbone LLM=GPT-4o-mini, Validation budget=502026.03 | 93.9 | |
| FLExBase Model=Gemma-12B, Model Family=Gemma-3 Instruct2026.01 | 93.8 | |
| SCBase Model=Qwen-72B, Model Family=Qwen-2.5 Instruct2026.01 | 93.6 | |
| AFlowBackbone=GPT-4o-mini2025.11 | 93.5 | |
| CoTBase Model=Gemma-12B, Model Family=Gemma-3 Instruct2026.01 | 93.4 | |
| MegaScienceData Volume=1.25M2026.03 | 93.4 | |
| AFlowBackbone LLM=GPT-4o-mini, Validation budget=502026.03 | 93.36 | |
| GPT-4o-mini (Random_diff)Backbone=GPT-4o-mini, Example Selection=Random_diff2024.04 | 93.3 | |
| GPT-4o-mini (Random_bio)Backbone=GPT-4o-mini, Example Selection=Random_bio2024.04 | 93.2 | |
| SRBase Model=Gemma-12B, Model Family=Gemma-3 Instruct2026.01 | 93.1 | |
| Jet-RLModel=Qwen3-8B-Base, Rollout Length=8k, Training Dataset (RL)=GSM8k + MATH2026.01 | 93.1 | |
| Self-ExaminationModel=GPT-4o-mini2025.05 | 93 | |
| ICDBackbone=Gemma-2-9B-IT2025.05 | 93 | |
| No DefenseBackbone=Qwen2.5-7B-Instruct2025.05 | 93 | |
| Self-ReminderBackbone=Qwen2.5-7B-Instruct2025.05 | 93 | |
| Self-ExaminationBackbone=Qwen2.5-7B-Instruct2025.05 | 93 | |
| SAGEBackbone=Qwen2.5-7B-Instruct2025.05 | 93 | |
| SCBase Model=Qwen-7B, Model Family=Qwen-2.5 Instruct2026.01 | 93 | |
| JudgeFlowBackbone LLM=GPT-4o-mini, System Category=Autonomous Multi-agent2026.02 | 93 | |
| TSBH (Pass@5)Model=Llama3.2-3B2026.04 | 93 | |
| ROASTModel=Qwen3-8B2026.02 | 92.95 | |
| RAGBase Model=Gemma-12B, Model Family=Gemma-3 Instruct2026.01 | 92.9 | |
| BF16Model=Qwen3-8B-Base, Rollout Length=8k, Training Dataset (RL)=GSM8k + MATH2026.01 | 92.9 | |
| MIPROBackbone LLM=GPT-4o-mini, Validation budget=502026.03 | 92.8 | |
| Fast-MathData Volume=7.9K2026.03 | 92.8 | |
| Raiden-DeepSeek-R1Data Volume=62K2026.03 | 92.72 | |
| QWQ-LongCoTData Volume=130K2026.03 | 92.72 | |
| RESP w/o re-genPruning Ratio=30%, Backbone=Qwen3-8B, prompt=8-shot2025.12 | 92.65 | |
| InstinctBackbone LLM=GPT-4o-mini, Validation budget=502026.03 | 92.64 | |
| BaselineModel=Qwen3-8B2026.02 | 92.61 | |
| GPT-4o-mini (Random_same)Backbone=GPT-4o-mini, Example Selection=Random_same2024.04 | 92.6 | |
| SCBase Model=Qwen-32B, Model Family=Qwen-2.5 Instruct2026.01 | 92.5 | |
| Few-shot-20Model=Qwen3-8B, Few-shot=202026.02 | 92.46 | |
| MATHFusionData Volume=74732026.03 | 92.42 | |
| MermaidFlowBackbone LLM=GPT-4o-mini, System Category=Autonomous Multi-agent2026.02 | 92.4 | |
| TraDo 8BPost-train=SFT+RL, Model Type=Diffusion, Number of shots=4, Evaluation Protocol=same protocol (*)2026.01 | 92.3 | |
| Full AttentionAttention complexity=O(L^2), Power (p)=12026.01 | 92.2 | |
| LIMO2Data Volume=8002026.03 | 92.19 | |
| CoTBase Model=Qwen-32B, Model Family=Qwen-2.5 Instruct2026.01 | 92.1 | |
| TATAModel=Qwen2.5-14B2025.02 | 92.1 | |
| BF16-Train-FP8-RolloutModel=Qwen3-8B-Base, Rollout Length=8k, Training Dataset (RL)=GSM8k + MATH2026.01 | 92.1 | |
| No DefenseBackbone=Gemma-2-9B-IT2025.05 | 92 | |
| ICDBackbone=Qwen2.5-7B-Instruct2025.05 | 92 | |
| Chain-of-ThoughtBackbone=Gemma-3 (27B), Reasoning Strategy=CoT2026.02 | 92 | |
| CoT-SCBackbone=GPT-4o-mini, n=52025.11 | 92 |