Language Modeling on Nemotron-CC 25M held-out tokens
9.58PerplexityMoE Transformer w/ dynamic conv. (low-rank)
Evaluation results
| Method | Links | |
|---|---|---|
| MoE Transformer w/ dynamic conv. (low-rank)Train Tokens=100B, Params=6.80B, Conv. Location=QKV2026.06 | 9.58 | |
| MoE Transformer w/ dynamic conv. (head-wise)Train Tokens=100B, Params=6.80B, Conv. Location=QKV2026.06 | 9.65 | |
| MoE Transformer w/ static conv.Train Tokens=100B, Params=6.77B, Conv. Location=QKV2026.06 | 9.74 | |
| MoE TransformerTrain Tokens=100B, Params=6.77B, Conv. Location=–2026.06 | 9.86 | |
| Transformer w/ dynamic conv. (low-rank)Train Tokens=100B, Params=1.88B, Conv. Location=all linear2026.06 | 10.95 | |
| Transformer w/ dynamic conv. (low-rank)Train Tokens=100B, Params=1.88B, Conv. Location=QKV2026.06 | 11.24 | |
| Transformer w/ dynamic conv. (head-wise)Train Tokens=100B, Params=1.88B, Conv. Location=QKV2026.06 | 11.41 | |
| Transformer w/ static conv.Train Tokens=100B, Params=1.83B, Conv. Location=QKV2026.06 | 11.5 | |
| Transformer w/ more params (wider FFN)Train Tokens=100B, Params=1.87B, Conv. Location=–2026.06 | 11.67 | |
| TransformerTrain Tokens=100B, Params=1.82B, Conv. Location=–2026.06 | 11.71 | |
| Transformer w/ dynamic conv. (low-rank)Train Tokens=15B, Params=319.0M, Conv. Location=all linear2026.06 | 17.42 | |
| Gated DeltaNet w/ dynamic conv. (low-rank)Train Tokens=15B, Params=309.5M, Conv. Location=QKV2026.06 | 17.95 | |
| Transformer w/ dynamic conv. (low-rank)Train Tokens=15B, Params=311.8M, Conv. Location=QKV2026.06 | 18.01 | |
| Gated DeltaNet w/ dynamic conv. (head-wise)Train Tokens=15B, Params=309.6M, Conv. Location=QKV2026.06 | 18.03 | |
| Transformer w/ dynamic conv. (head-wise)Train Tokens=15B, Params=311.7M, Conv. Location=QKV2026.06 | 18.22 | |
| Transformer w/ static conv.Train Tokens=15B, Params=305.4M, Conv. Location=QKV2026.06 | 18.66 | |
| Mamba-2 w/ dynamic conv. (head-wise)Train Tokens=15B, Params=309.8M, Conv. Location=QKV2026.06 | 18.69 | |
| Mamba-2 w/ dynamic conv. (low-rank)Train Tokens=15B, Params=309.8M, Conv. Location=QKV2026.06 | 18.72 | |
| Gated DeltaNet w/ static conv.Train Tokens=15B, Params=305.4M, Conv. Location=QKV2026.06 | 18.75 | |
| Gated DeltaNet (w/o conv.)Train Tokens=15B, Params=305.2M, Conv. Location=–2026.06 | 18.93 | |
| Transformer w/ more params (wider FFN)Train Tokens=15B, Params=311.5M, Conv. Location=–2026.06 | 18.99 | |
| TransformerTrain Tokens=15B, Params=305.2M, Conv. Location=–2026.06 | 19.12 | |
| Mamba-2 w/ static conv.Train Tokens=15B, Params=306.4M, Conv. Location=QKV2026.06 | 19.3 | |
| Mamba-2 (w/o conv.)Train Tokens=15B, Params=306.2M, Conv. Location=–2026.06 | 20.26 |