Mathematical Reasoning on AIME '24 (Accuracy, Avg, Speedup)
93.3Accuracyall-attention
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| all-attentionSpeedup @32k=1.0x, Speedup @16k=1.0x2026.04 | 93.3 | — | — | |
| Nemotron-3-Nano 30BSpeedup @32k=4.09x, Speedup @16k=2.8x2026.04 | 93.3 | — | — | |
| Idealized|All–18Speedup @32k=1.99x, Speedup @16k=1.1x2026.04 | 90 | — | — | |
| Apriel-1.6Speedup @32k=1.0x, Speedup @16k=1.0x2026.04 | 90 | — | — | |
| Qwen-3.5 27BSpeedup @32k=0.55x, Speedup @16k=0.5x2026.04 | 90 | — | — | |
| Reg|Lklhd–18Speedup @32k=4.76x, Speedup @16k=2.2x2026.04 | 86.7 | — | — | |
| Reg|Lklhd–26Speedup @32k=2.85x, Speedup @16k=1.5x2026.04 | 86.7 | — | — | |
| Idealized|Lklhd–6Speedup @32k=6.2x, Speedup @16k=2.4x2026.04 | 83.3 | — | — | |
| Idealized|All–6Speedup @32k=6.13x, Speedup @16k=2.5x2026.04 | 83.3 | — | — | |
| Apriel-H1 15BSpeedup @32k=1.97x, Speedup @16k=1.9x2026.04 | 80 | — | — | |
| Reg|Lklhd–10Speedup @32k=10.69x, Speedup @16k=4.2x2026.04 | 76.7 | — | — | |
| Reg|Lklhd–13Speedup @32k=6.9x, Speedup @16k=2.7x2026.04 | 76.7 | — | — | |
| OLMo-Hybrid-Think 7BSpeedup @32k=2.51x, Speedup @16k=2.1x2026.04 | 66.7 | — | — | |
| Falcon-H1R 7BSpeedup @32k=4.61x, Speedup @16k=3.4x2026.04 | 66.7 | — | — | |
| GRPO + ARROLBackbone=Qwen-3-8B-Base, Method=GRPO + ARROL2026.03 | 66.67 | 59.53 | 1.62 | |
| S1: Distil. Idealized|All–6Speedup @32k=6.13x, Speedup @16k=2.5x2026.04 | 63.3 | — | — | |
| Nemotron-Nano 12B v2Speedup @32k=5.85x, Speedup @16k=4.3x2026.04 | 60 | — | — | |
| GRPOBackbone=Qwen-3-8B-Base, Method=GRPO2026.03 | 56.67 | 56.66 | — | |
| GRPO + ARROLBackbone=Qwen-3-4B-Base, Method=GRPO + ARROL2026.03 | 40 | 53.54 | 1.63 | |
| GRPOBackbone=Qwen-3-4B-Base, Method=GRPO2026.03 | 36.67 | 51.01 | — | |
| Soft-ORModel pair=Llama-70B → 8B, Selection Percentage=50%2026.04 | 26 | — | — | |
| Soft-ORModel pair=Qwen3-8B → 4B, Selection Percentage=50%2026.04 | 25.7 | — | — | |
| Entropy-onlyModel pair=Llama-70B → 8B, Selection Percentage=50%2026.04 | 25.3 | — | — | |
| Soft-ORModel pair=Qwen3-8B → 4B, Selection Percentage=20%2026.04 | 24.5 | — | — | |
| Entropy-onlyModel pair=Qwen3-8B → 4B, Selection Percentage=50%2026.04 | 23.8 | — | — | |
| GRPO + ARROLBackbone=Qwen-3-1.7B-Base, Method=GRPO + ARROL2026.03 | 23.33 | 37.09 | 1.61 | |
| Entropy-onlyModel pair=Qwen3-8B → 4B, Selection Percentage=20%2026.04 | 22.5 | — | — | |
| BaselineModel pair=Qwen3-8B → 4B, Selection Percentage=100%2026.04 | 21.9 | — | — | |
| BaselineModel pair=Llama-70B → 8B, Selection Percentage=100%2026.04 | 21.5 | — | — | |
| Soft-ORModel pair=Llama-70B → 8B, Selection Percentage=20%2026.04 | 21 | — | — | |
| GRPOBackbone=Qwen-3-1.7B-Base, Method=GRPO2026.03 | 20 | 34.79 | — | |
| Entropy-onlyModel pair=Llama-70B → 8B, Selection Percentage=20%2026.04 | 18.8 | — | — | |
| GRPO + ARROLBackbone=Llama-3.2-1B-Instruct, Method=GRPO + ARROL2026.03 | 16.67 | 17.49 | 1.67 | |
| GRPOBackbone=Llama-3.2-1B-Instruct, Method=GRPO2026.03 | 13.33 | 14.63 | — | |
| Soft-ORModel pair=Qwen2.5-14B → 1.5B, Selection Percentage=20%2026.04 | 5 | — | — | |
| Entropy-onlyModel pair=Qwen2.5-14B → 1.5B, Selection Percentage=20%2026.04 | 4.6 | — | — | |
| Soft-ORModel pair=Qwen2.5-14B → 1.5B, Selection Percentage=50%2026.04 | 3.8 | — | — | |
| Entropy-onlyModel pair=Qwen2.5-14B → 1.5B, Selection Percentage=50%2026.04 | 3.3 | — | — | |
| BaselineModel pair=Qwen2.5-14B → 1.5B, Selection Percentage=100%2026.04 | 2.4 | — | — | |
| NebulaExp-reasoning-SFTBackbone=Qwen3-8B-base2026.06 | — | 80.83 | — | |
| Qwen3-14BBackbone=Qwen3-14B2026.06 | — | 79.3 | — | |
| Qwen3-8B-thinkingBackbone=Qwen3-8B, Mode=thinking2026.06 | — | 76 | — | |
| QWQ-32BBackbone=QWQ-32B2026.06 | — | 79.5 | — |