Mathematical Reasoning on GSM8K (test) (Accuracy, 95% CI)
96.5Accuracy (GSM8K)Seed
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| SeedBootstrap resampling=N=10002026.05 | 96.5 | — | — | |
| SPEARBootstrap resampling=N=10002026.05 | 96.5 | 0.955 | — | |
| GEPABootstrap resampling=N=10002026.05 | 96 | 0.948 | — | |
| TextGradBootstrap resampling=N=10002026.05 | 95.6 | 0.945 | — | |
| Self-ConsistencyBackbone=Qwen2.5-7B-Instruct, Method Category=Non-RL Baselines2026.05 | 92.4 | — | 62.42 | |
| CoTBackbone=Qwen2.5-7B-Instruct, Method Category=Non-RL Baselines2026.05 | 91.6 | — | 67.83 | |
| Self-ConsistencyBackbone=Phi-3 Mini 4K Instruct, Method Category=Non-RL Baselines2026.05 | 90.37 | — | 52.32 | |
| TRACERBackbone=Qwen2.5-7B-Instruct, Method Category=Non-RL/RL Baseline Representative2026.05 | 89.01 | — | 61.85 | |
| Sparse MADBackbone=Qwen2.5-7B-Instruct, Method Category=Non-RL Baselines2026.05 | 88.78 | — | 55.8 | |
| MoABackbone=Qwen2.5-7B-Instruct, Method Category=Non-RL Baselines2026.05 | 88.7 | — | 53.39 | |
| MADBackbone=Qwen2.5-7B-Instruct, Method Category=Non-RL Baselines2026.05 | 88.62 | — | 56.11 | |
| Single-Agent GRPOBackbone=Qwen2.5-7B-Instruct, Method Category=RL Baselines2026.05 | 88.25 | — | 46.49 | |
| TRACERBackbone=Phi-3 Mini 4K Instruct, Method Category=Non-RL/RL Baseline Representative2026.05 | 88.02 | — | 50.27 | |
| Single-Agent GSPOBackbone=Phi-3 Mini 4K Instruct, Method Category=RL Baselines2026.05 | 87.95 | — | 45.72 | |
| Single-Agent GSPOBackbone=Qwen2.5-7B-Instruct, Method Category=RL Baselines2026.05 | 87.41 | — | 45.63 | |
| Single-Agent GRPOBackbone=Phi-3 Mini 4K Instruct, Method Category=RL Baselines2026.05 | 86.81 | — | 46.63 | |
| FullKVBackbone=QWEN2.5-14B-INSTRUCT, 50-shot prompting=true, KV-cache budget=uncompressed2026.06 | 86.7 | — | — | |
| Sparse MADBackbone=Phi-3 Mini 4K Instruct, Method Category=Non-RL Baselines2026.05 | 84.69 | — | 51.13 | |
| SEKVBackbone=QWEN2.5-14B-INSTRUCT, 50-shot prompting=true, KV-cache budget=10% GPU-resident2026.06 | 84.5 | — | — | |
| MAPoRLBackbone=Phi-3 Mini 4K Instruct, Method Category=RL Baselines2026.05 | 84.15 | — | 48.93 | |
| MADBackbone=Phi-3 Mini 4K Instruct, Method Category=Non-RL Baselines2026.05 | 84.08 | — | 51.1 | |
| MAPoRLBackbone=Qwen2.5-7B-Instruct, Method Category=RL Baselines2026.05 | 83.85 | — | 38.19 | |
| CoTBackbone=Phi-3 Mini 4K Instruct, Method Category=Non-RL Baselines2026.05 | 82.5 | — | 52.2 | |
| FullKVBackbone=LLAMA-3.1-8B-INSTRUCT, 50-shot prompting=true, KV-cache budget=uncompressed2026.06 | 82.4 | — | — | |
| SentenceKVBackbone=QWEN2.5-14B-INSTRUCT, 50-shot prompting=true, KV-cache budget=10% GPU-resident2026.06 | 82.1 | — | — | |
| ChunkKVBackbone=QWEN2.5-14B-INSTRUCT, 50-shot prompting=true, KV-cache budget=10% GPU-resident2026.06 | 80.7 | — | — | |
| MoABackbone=Phi-3 Mini 4K Instruct, Method Category=Non-RL Baselines2026.05 | 80.44 | — | 47.26 | |
| PyramidKVBackbone=QWEN2.5-14B-INSTRUCT, 50-shot prompting=true, KV-cache budget=10% GPU-resident2026.06 | 79.4 | — | — | |
| SEKVBackbone=LLAMA-3.1-8B-INSTRUCT, 50-shot prompting=true, KV-cache budget=10% GPU-resident2026.06 | 79.3 | — | — | |
| SnapKVBackbone=QWEN2.5-14B-INSTRUCT, 50-shot prompting=true, KV-cache budget=10% GPU-resident2026.06 | 77.9 | — | — | |
| SentenceKVBackbone=LLAMA-3.1-8B-INSTRUCT, 50-shot prompting=true, KV-cache budget=10% GPU-resident2026.06 | 76.8 | — | — | |
| ChunkKVBackbone=LLAMA-3.1-8B-INSTRUCT, 50-shot prompting=true, KV-cache budget=10% GPU-resident2026.06 | 74.9 | — | — | |
| FullKVBackbone=MISTRAL-7B-INSTRUCT, 50-shot prompting=true, KV-cache budget=uncompressed2026.06 | 74.6 | — | — | |
| MAGRPOBackbone=Phi-3 Mini 4K Instruct, Method Category=RL Baselines2026.05 | 72.52 | — | 38.52 | |
| SEKVBackbone=MISTRAL-7B-INSTRUCT, 50-shot prompting=true, KV-cache budget=10% GPU-resident2026.06 | 72.5 | — | — | |
| SentenceKVBackbone=MISTRAL-7B-INSTRUCT, 50-shot prompting=true, KV-cache budget=10% GPU-resident2026.06 | 70.4 | — | — | |
| PyramidKVBackbone=LLAMA-3.1-8B-INSTRUCT, 50-shot prompting=true, KV-cache budget=10% GPU-resident2026.06 | 70.3 | — | — | |
| MAGRPOBackbone=Qwen2.5-7B-Instruct, Method Category=RL Baselines2026.05 | 69.6 | — | 34.73 | |
| FullKVBackbone=LLAMA-3-8B, 50-shot prompting=true, KV-cache budget=uncompressed2026.06 | 69.4 | — | — | |
| ChunkKVBackbone=MISTRAL-7B-INSTRUCT, 50-shot prompting=true, KV-cache budget=10% GPU-resident2026.06 | 69.3 | — | — | |
| SnapKVBackbone=LLAMA-3.1-8B-INSTRUCT, 50-shot prompting=true, KV-cache budget=10% GPU-resident2026.06 | 68.2 | — | — | |
| PyramidKVBackbone=MISTRAL-7B-INSTRUCT, 50-shot prompting=true, KV-cache budget=10% GPU-resident2026.06 | 68.1 | — | — | |
| SEKVBackbone=LLAMA-3-8B, 50-shot prompting=true, KV-cache budget=10% GPU-resident2026.06 | 67.8 | — | — | |
| SnapKVBackbone=MISTRAL-7B-INSTRUCT, 50-shot prompting=true, KV-cache budget=10% GPU-resident2026.06 | 66.7 | — | — | |
| SentenceKVBackbone=LLAMA-3-8B, 50-shot prompting=true, KV-cache budget=10% GPU-resident2026.06 | 65.4 | — | — | |
| ChunkKVBackbone=LLAMA-3-8B, 50-shot prompting=true, KV-cache budget=10% GPU-resident2026.06 | 64.6 | — | — | |
| PyramidKVBackbone=LLAMA-3-8B, 50-shot prompting=true, KV-cache budget=10% GPU-resident2026.06 | 63.2 | — | — | |
| SnapKVBackbone=LLAMA-3-8B, 50-shot prompting=true, KV-cache budget=10% GPU-resident2026.06 | 61.7 | — | — | |
| FullKVBackbone=LLAMA-3.2-3B-INSTRUCT, 50-shot prompting=true, KV-cache budget=uncompressed2026.06 | 60.8 | — | — | |
| SEKVBackbone=LLAMA-3.2-3B-INSTRUCT, 50-shot prompting=true, KV-cache budget=10% GPU-resident2026.06 | 58.4 | — | — | |
| SentenceKVBackbone=LLAMA-3.2-3B-INSTRUCT, 50-shot prompting=true, KV-cache budget=10% GPU-resident2026.06 | 56.1 | — | — | |
| ChunkKVBackbone=LLAMA-3.2-3B-INSTRUCT, 50-shot prompting=true, KV-cache budget=10% GPU-resident2026.06 | 55.6 | — | — | |
| PyramidKVBackbone=LLAMA-3.2-3B-INSTRUCT, 50-shot prompting=true, KV-cache budget=10% GPU-resident2026.06 | 54.2 | — | — | |
| SnapKVBackbone=LLAMA-3.2-3B-INSTRUCT, 50-shot prompting=true, KV-cache budget=10% GPU-resident2026.06 | 53.4 | — | — |