STEM Problem Solving on S-GPQA, GPQA-D, GSM8k, and Math
25.81S-GPQAGatedNorm
Evaluation Results
| Method | Links | ||||
|---|---|---|---|---|---|
| GatedNormArchitecture=24B-A3B, Precision=BF16, Training Tokens=500B2026.01 | 25.81 | 33.87 | 82.26 | 62.7 | |
| Gated AttentionArchitecture=24B-A3B, Precision=BF16, Training Tokens=500B2026.01 | 25.64 | 31.66 | 79.45 | 53.92 | |
| GatedNormArchitecture=24B-A3B, Precision=W4A4, Training Tokens=500B2026.01 | 25.59 | 32.7 | 81.35 | 59.4 | |
| GatedNormArchitecture=24B-A3B, Precision=W4A4+SQ, Training Tokens=500B2026.01 | 25.11 | 33.62 | 80.59 | 61.88 | |
| Gated AttentionArchitecture=24B-A3B, Precision=W4A4, Training Tokens=500B2026.01 | 24.73 | 31.98 | 76.38 | 52.66 | |
| Gated AttentionArchitecture=24B-A3B, Precision=W4A4+SQ, Training Tokens=500B2026.01 | 24.68 | 32.77 | 77.45 | 52.82 | |
| PreAffineArchitecture=24B-A3B, Precision=BF16, Training Tokens=500B2026.01 | 24.34 | 32.64 | 82.07 | 52.62 | |
| PreAffineArchitecture=24B-A3B, Precision=W4A4, Training Tokens=500B2026.01 | 24.14 | 31.6 | 78.7 | 49.5 | |
| PreAffineArchitecture=24B-A3B, Precision=W4A4+SQ, Training Tokens=500B2026.01 | 23.77 | 31.66 | 79.42 | 49.05 | |
| GatedNormArchitecture=MoE-7B-A2B, Precision=BF16, Training Tokens=1.2T2026.01 | 19.05 | 29.9 | 68.04 | 46.66 | |
| PreAffineArchitecture=MoE-7B-A2B, Precision=BF16, Training Tokens=1.2T2026.01 | 18.75 | 27.4 | 67.1 | 44.26 | |
| Gated AttentionArchitecture=MoE-7B-A2B, Precision=BF16, Training Tokens=1.2T2026.01 | 16.39 | 26.93 | 66.22 | 45.36 |