Language Modeling on LAMBADA (PPL)
4.11Perplexity (PPL)K-stage RK
Evaluation results
| Method | Links | |
|---|---|---|
| K-stage RKSolver choice=Runge–Kutta, K=22026.05 | 4.11 | |
| Baseline2026.05 | 4.12 | |
| Heun K=1Solver choice=Heun, K=12026.05 | 4.12 | |
| MoE Transformer w/ dynamic conv. (low-rank)Train Tokens=100B, Params=6.80B, Conv. Location=QKV2026.06 | 10.92 | |
| MoE Transformer w/ static conv.Train Tokens=100B, Params=6.77B, Conv. Location=QKV2026.06 | 11.2 | |
| MoE TransformerTrain Tokens=100B, Params=6.77B, Conv. Location=–2026.06 | 11.55 | |
| MoE Transformer w/ dynamic conv. (head-wise)Train Tokens=100B, Params=6.80B, Conv. Location=QKV2026.06 | 11.61 | |
| Transformer w/ dynamic conv. (low-rank)Train Tokens=100B, Params=1.88B, Conv. Location=all linear2026.06 | 12.51 | |
| Transformer w/ dynamic conv. (low-rank)Train Tokens=100B, Params=1.88B, Conv. Location=QKV2026.06 | 15.43 | |
| Transformer w/ static conv.Train Tokens=100B, Params=1.83B, Conv. Location=QKV2026.06 | 16.21 | |
| Transformer w/ dynamic conv. (head-wise)Train Tokens=100B, Params=1.88B, Conv. Location=QKV2026.06 | 16.34 | |
| TransformerTrain Tokens=100B, Params=1.82B, Conv. Location=–2026.06 | 17.28 | |
| Transformer w/ more params (wider FFN)Train Tokens=100B, Params=1.87B, Conv. Location=–2026.06 | 18.49 | |
| Transformer w/ dynamic conv. (low-rank)Train Tokens=15B, Params=319.0M, Conv. Location=all linear2026.06 | 46.13 | |
| Gated DeltaNet w/ dynamic conv. (low-rank)Train Tokens=15B, Params=309.5M, Conv. Location=QKV2026.06 | 50.56 | |
| Transformer w/ dynamic conv. (low-rank)Train Tokens=15B, Params=311.8M, Conv. Location=QKV2026.06 | 56.66 | |
| Transformer w/ dynamic conv. (head-wise)Train Tokens=15B, Params=311.7M, Conv. Location=QKV2026.06 | 58.54 | |
| Gated DeltaNet w/ dynamic conv. (head-wise)Train Tokens=15B, Params=309.6M, Conv. Location=QKV2026.06 | 59.49 | |
| Mamba-2 w/ dynamic conv. (head-wise)Train Tokens=15B, Params=309.8M, Conv. Location=QKV2026.06 | 65.8 | |
| Gated DeltaNet w/ static conv.Train Tokens=15B, Params=305.4M, Conv. Location=QKV2026.06 | 67.63 | |
| Transformer w/ more params (wider FFN)Train Tokens=15B, Params=311.5M, Conv. Location=–2026.06 | 68.05 | |
| Transformer w/ static conv.Train Tokens=15B, Params=305.4M, Conv. Location=QKV2026.06 | 69.64 | |
| Gated DeltaNet (w/o conv.)Train Tokens=15B, Params=305.2M, Conv. Location=–2026.06 | 69.9 | |
| Mamba-2 w/ dynamic conv. (low-rank)Train Tokens=15B, Params=309.8M, Conv. Location=QKV2026.06 | 71.12 | |
| TransformerTrain Tokens=15B, Params=305.2M, Conv. Location=–2026.06 | 76.62 | |
| Mamba-2 (w/o conv.)Train Tokens=15B, Params=306.2M, Conv. Location=–2026.06 | 80.81 | |
| Mamba-2 w/ static conv.Train Tokens=15B, Params=306.4M, Conv. Location=QKV2026.06 | 83.5 |