Common-sense Reasoning on lm-eval-harness 11 tasks
63.430-shot AccuracyMoE Transformer w/ dynamic conv. (head-wise)
Evaluation results
| Method | Links | |
|---|---|---|
| MoE Transformer w/ dynamic conv. (head-wise)Train Tokens=100B, Params=6.80B, Conv. Location=QKV, Zero-shot=true2026.06 | 63.43 | |
| MoE Transformer w/ dynamic conv. (low-rank)Train Tokens=100B, Params=6.80B, Conv. Location=QKV, Zero-shot=true2026.06 | 63.42 | |
| MoE Transformer w/ static conv.Train Tokens=100B, Params=6.77B, Conv. Location=QKV, Zero-shot=true2026.06 | 62.97 | |
| MoE TransformerTrain Tokens=100B, Params=6.77B, Conv. Location=–, Zero-shot=true2026.06 | 62.46 | |
| Transformer w/ dynamic conv. (low-rank)Train Tokens=100B, Params=1.88B, Conv. Location=all linear, Zero-shot=true2026.06 | 60.7 | |
| Transformer w/ dynamic conv. (low-rank)Train Tokens=100B, Params=1.88B, Conv. Location=QKV, Zero-shot=true2026.06 | 59.7 | |
| Transformer w/ static conv.Train Tokens=100B, Params=1.83B, Conv. Location=QKV, Zero-shot=true2026.06 | 58.94 | |
| Transformer w/ dynamic conv. (head-wise)Train Tokens=100B, Params=1.88B, Conv. Location=QKV, Zero-shot=true2026.06 | 58.46 | |
| TransformerTrain Tokens=100B, Params=1.82B, Conv. Location=–, Zero-shot=true2026.06 | 58.35 | |
| Transformer w/ more params (wider FFN)Train Tokens=100B, Params=1.87B, Conv. Location=–, Zero-shot=true2026.06 | 58.23 | |
| Gated DeltaNet w/ dynamic conv. (low-rank)Train Tokens=15B, Params=309.5M, Conv. Location=QKV, Zero-shot=true2026.06 | 49.22 | |
| Transformer w/ dynamic conv. (low-rank)Train Tokens=15B, Params=311.8M, Conv. Location=QKV, Zero-shot=true2026.06 | 48.9 | |
| Transformer w/ dynamic conv. (low-rank)Train Tokens=15B, Params=319.0M, Conv. Location=all linear, Zero-shot=true2026.06 | 48.81 | |
| Transformer w/ dynamic conv. (head-wise)Train Tokens=15B, Params=311.7M, Conv. Location=QKV, Zero-shot=true2026.06 | 47.61 | |
| Mamba-2 w/ dynamic conv. (low-rank)Train Tokens=15B, Params=309.8M, Conv. Location=QKV, Zero-shot=true2026.06 | 47.34 | |
| Gated DeltaNet w/ dynamic conv. (head-wise)Train Tokens=15B, Params=309.6M, Conv. Location=QKV, Zero-shot=true2026.06 | 47.27 | |
| TransformerTrain Tokens=15B, Params=305.2M, Conv. Location=–, Zero-shot=true2026.06 | 47.26 | |
| Mamba-2 w/ dynamic conv. (head-wise)Train Tokens=15B, Params=309.8M, Conv. Location=QKV, Zero-shot=true2026.06 | 47.24 | |
| Gated DeltaNet (w/o conv.)Train Tokens=15B, Params=305.2M, Conv. Location=–, Zero-shot=true2026.06 | 46.93 | |
| Transformer w/ static conv.Train Tokens=15B, Params=305.4M, Conv. Location=QKV, Zero-shot=true2026.06 | 46.86 | |
| Gated DeltaNet w/ static conv.Train Tokens=15B, Params=305.4M, Conv. Location=QKV, Zero-shot=true2026.06 | 46.76 | |
| Transformer w/ more params (wider FFN)Train Tokens=15B, Params=311.5M, Conv. Location=–, Zero-shot=true2026.06 | 46.64 | |
| Mamba-2 (w/o conv.)Train Tokens=15B, Params=306.2M, Conv. Location=–, Zero-shot=true2026.06 | 46.41 | |
| Mamba-2 w/ static conv.Train Tokens=15B, Params=306.4M, Conv. Location=QKV, Zero-shot=true2026.06 | 45.78 |