Language Modeling on SlimPajama data-restricted
24.35Average PerplexityTANDEM
Evaluation results
| Method | Links | ||
|---|---|---|---|
| TANDEMModel Size=1B2026.06 | 24.35 | 77 | |
| TANDEMModel Size=410M2026.06 | 25 | 57 | |
| Skill-ItModel Size=1B2026.06 | 27.15 | 80 | |
| DoGEModel Size=410M2026.06 | 27.45 | 172 | |
| Skill-ItModel Size=410M2026.06 | 27.7 | 59 | |
| AioliModel Size=1B2026.06 | 28.07 | 93 | |
| TANDEMModel Size=160M2026.06 | 28.07 | 26 | |
| AioliModel Size=410M2026.06 | 28.79 | 64 | |
| Skill-ItModel Size=160M2026.06 | 29.24 | 28 | |
| UniformModel Size=410M2026.06 | 29.59 | — | |
| UniformModel Size=1B2026.06 | 29.91 | — | |
| DoGEModel Size=160M2026.06 | 30.1 | 65 | |
| AioliModel Size=160M2026.06 | 31.19 | 36 | |
| UniformModel Size=160M2026.06 | 31.53 | — | |
| DoReMiModel Size=160M2026.06 | 36.91 | 16 | |
| DoReMiModel Size=410M2026.06 | 54.61 | 31 | |
| DoReMiModel Size=1B2026.06 | 56.53 | 41 |