Language Modeling on SlimPajama-672B (val)
8.09Validation PerplexityKLA
Evaluation results
| Method | Links | |
|---|---|---|
| KLAModel Scale=0.4B, Training Budget=1B-token, Training Context=4K2026.05 | 8.09 | |
| DeltaNetModel Scale=0.4B, Training Budget=1B-token, Training Context=4K2026.05 | 8.26 | |
| GLAModel Scale=0.4B, Training Budget=1B-token, Training Context=4K2026.05 | 8.42 | |
| GDNModel Scale=0.4B, Training Budget=1B-token, Training Context=4K2026.05 | 8.5 | |
| LonghornModel Scale=0.4B, Training Budget=1B-token, Training Context=4K2026.05 | 8.79 | |
| Mamba2Model Scale=0.4B, Training Budget=1B-token, Training Context=4K2026.05 | 9.41 |