Language Modeling on SlimPajama (val)
3.09PerplexityDoGraph
Evaluation Results
| Method | Links | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| DoGraphBackbone=GPT-2 Medium2026.04 | 3.09 | — | — | — | — | — | — | — | — | |
| DYNAMIC LOSS-BASEDBackbone=GPT-2 Medium2026.04 | 3.1 | — | — | — | — | — | — | — | — | |
| DoReMiBackbone=GPT-2 Medium2026.04 | 3.3 | — | — | — | — | — | — | — | — | |
| DOGEBackbone=GPT-2 Medium2026.04 | 3.31 | — | — | — | — | — | — | — | — | |
| UniformBackbone=GPT-2 Medium2026.04 | 4.13 | — | — | — | — | — | — | — | — | |
| Data Mixing LawBackbone=GPT-2 Medium2026.04 | 4.5 | — | — | — | — | — | — | — | — | |
| RegMixBackbone=GPT-2 Medium2026.04 | 4.51 | — | — | — | — | — | — | — | — | |
| Learnable ResFormer plusParameters=468M, Training Tokens=20B, Backbone=Llama-like, Sequence Length=20482024.10 | 23.2 | — | — | — | — | — | — | — | — | |
| Learnable ResFormerParameters=468M, Training Tokens=20B, Backbone=Llama-like, Sequence Length=20482024.10 | 23.7 | — | — | — | — | — | — | — | — | |
| Identity ResFormerParameters=468M, Training Tokens=20B, Backbone=Llama-like, Sequence Length=20482024.10 | 23.8 | — | — | — | — | — | — | — | — | |
| DenseFormerParameters=468M, Training Tokens=20B, Backbone=Llama-like, Sequence Length=20482024.10 | 24 | — | — | — | — | — | — | — | — | |
| NeuTRENOParameters=468M, Training Tokens=20B, Backbone=Llama-like, Sequence Length=20482024.10 | 24.3 | — | — | — | — | — | — | — | — | |
| TransformerParameters=468M, Training Tokens=20B, Backbone=Llama-like, Sequence Length=20482024.10 | 24.8 | — | — | — | — | — | — | — | — | |
| BaselineTraining tokens=6B2026.03 | — | 4.217 | 4.278 | 4.532 | 3.402 | 3.546 | 2.64 | 3.508 | 4.778 | |
| BaselineTraining tokens=30B2026.03 | — | 3.584 | 3.723 | 3.505 | 2.85 | 3.215 | 3.163 | 4.54 | 5.329 | |
| DoReMiTraining tokens=6B2026.03 | — | 4.134 | 4.108 | 4.358 | 3.788 | 3.997 | 3.42 | 3.413 | 4.661 | |
| DoReMiTraining tokens=30B2026.03 | — | 3.562 | 3.731 | 3.503 | 2.706 | 2.985 | 2.973 | 4.441 | 5.214 | |
| ODMTraining tokens=6B2026.03 | — | 4.244 | 4.326 | 4.555 | 3.243 | 3.699 | 2.704 | 2.904 | 4.613 | |
| ODMTraining tokens=30B2026.03 | — | 3.429 | 3.598 | 3.519 | 2.382 | 2.713 | 2.255 | 3.487 | 4.746 |