Language Modeling on (val)
7.32PerplexityDAPT (nontoxic)
Evaluation results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| DAPT (nontoxic)Parameter Size=530B2022.02 | 7.32 | — | — | — | — | — | — | — | |
| SGEAT (augmented)Parameter Size=530B, Training Samples=50k, Training Epochs=32022.02 | 7.86 | — | — | — | — | — | — | — | |
| SGEAT (augmented)Parameter Size=530B, Training Samples=100k, Training Epochs=32022.02 | 7.92 | — | — | — | — | — | — | — | |
| DAPT (nontoxic)Parameter Size=8.3B2022.02 | 8.12 | — | — | — | — | — | — | — | |
| SGEAT (augmented)Parameter Size=8.3B2022.02 | 8.91 | — | — | — | — | — | — | — | |
| SGEAT (augmented)Parameter Size=530B, Training Samples=50k, Training Epochs=52022.02 | 9.63 | — | — | — | — | — | — | — | |
| DAPT (nontoxic)Parameter Size=1.3B2022.02 | 10.4 | — | — | — | — | — | — | — | |
| SGEAT (augmented)Parameter Size=1.3B2022.02 | 11.19 | — | — | — | — | — | — | — | |
| DAPT (nontoxic)Parameter Size=357M2022.02 | 13.33 | — | — | — | — | — | — | — | |
| Dynamic ExoFormerAnchor Source=External Anchor, Mixing Granularity=Elementwise, Normalization Strategy=Full2026.01 | 14.09 | — | — | — | — | — | — | — | |
| ExoFormerAnchor Source=External Anchor, Mixing Granularity=Elementwise, Normalization Strategy=Full2026.01 | 14.13 | — | — | — | — | — | — | — | |
| ExoFormerAnchor Source=External Anchor, Mixing Granularity=Headwise, Normalization Strategy=Full2026.01 | 14.14 | — | — | — | — | — | — | — | |
| NuResFormerAnchor Source=Internal Anchor, Mixing Granularity=Elementwise, Normalization Strategy=Full2026.01 | 14.15 | — | — | — | — | — | — | — | |
| ExoFormerAnchor Source=External Anchor, Mixing Granularity=Scalar, Normalization Strategy=Full2026.01 | 14.15 | — | — | — | — | — | — | — | |
| NuResFormerAnchor Source=Internal Anchor, Mixing Granularity=Headwise, Normalization Strategy=Full2026.01 | 14.17 | — | — | — | — | — | — | — | |
| NuResFormerAnchor Source=Internal Anchor, Mixing Granularity=Scalar, Normalization Strategy=Full2026.01 | 14.17 | — | — | — | — | — | — | — | |
| NuResFormerAnchor Source=Internal Anchor, Mixing Granularity=Elementwise, Normalization Strategy=Only Q/K Norms2026.01 | 14.21 | — | — | — | — | — | — | — | |
| NuResFormerAnchor Source=Internal Anchor, Mixing Granularity=Headwise, Normalization Strategy=Only Q/K Norms2026.01 | 14.21 | — | — | — | — | — | — | — | |
| NuResFormerAnchor Source=Internal Anchor, Mixing Granularity=Scalar, Normalization Strategy=Only Q/K Norms2026.01 | 14.22 | — | — | — | — | — | — | — | |
| NuResFormerAnchor Source=Internal Anchor, Mixing Granularity=Headwise, Normalization Strategy=No Norm, Mixing Path Components={V,G}2026.01 | 14.22 | — | — | — | — | — | — | — | |
| NuResFormerAnchor Source=Internal Anchor, Mixing Granularity=Elementwise, Normalization Strategy=No Norm, Mixing Path Components={V,G}2026.01 | 14.24 | — | — | — | — | — | — | — | |
| NuResFormerAnchor Source=Internal Anchor, Mixing Granularity=Scalar, Normalization Strategy=No Norm, Mixing Path Components={V,G}2026.01 | 14.24 | — | — | — | — | — | — | — | |
| Naïve CombinationAnchor Source=Baselines2026.01 | 14.25 | — | — | — | — | — | — | — | |
| ExoFormerAnchor Source=External Anchor, Mixing Granularity=Elementwise, Normalization Strategy=No Norm2026.01 | 14.3 | — | — | — | — | — | — | — | |
| ResFormer (Value Residual)Anchor Source=Baselines2026.01 | 14.32 | — | — | — | — | — | — | — | |
| SGEAT (augmented)Parameter Size=357M2022.02 | 14.39 | — | — | — | — | — | — | — | |
| Gated AttentionAnchor Source=Baselines2026.01 | 14.64 | — | — | — | — | — | — | — | |
| Base TransformerAnchor Source=Baselines2026.01 | 14.79 | — | — | — | — | — | — | — | |
| DAPT (nontoxic)Parameter Size=126M2022.02 | 17.97 | — | — | — | — | — | — | — | |
| SGEAT (augmented)Parameter Size=126M2022.02 | 19.55 | — | — | — | — | — | — | — | |
| GDNlong-context fine-tuning=false, metric=perplexity2026.05 | — | 42.9 | 43.5 | 43.8 | 47.4 | 49.4 | 48.4 | 47.5 | |
| KLAlong-context fine-tuning=false, metric=perplexity2026.05 | — | 39.1 | 39.8 | 40 | 43.6 | 45.3 | 44.5 | 43.6 | |
| Mamba2long-context fine-tuning=false, metric=perplexity2026.05 | — | 49.9 | 50.9 | 52.2 | 56.4 | 58.7 | 57.3 | 56.7 |