Language Modeling on FineWeb-Edu 500M-token (val)
2.221Valid LossFusedKV
Evaluation results
| Method | Links | |
|---|---|---|
| FusedKVModel Scale=1.5B2025.12 | 2.221 | |
| FusedKV-LiteModel Scale=1.5B2025.12 | 2.225 | |
| VanillaModel Scale=1.5B, Cache Mem.=12025.12 | 2.241 | |
| YOCOModel Scale=1.5B2025.12 | 2.244 | |
| GQAModel Scale=1.5B2025.12 | 2.245 | |
| CLAModel Scale=1.5B, Cache Mem.=1/22025.12 | 2.258 | |
| FusedKV-LiteModel Scale=650M2025.12 | 2.473 | |
| FusedKVModel Scale=650M2025.12 | 2.474 | |
| VanillaModel Scale=650M, Cache Mem.=12025.12 | 2.483 | |
| GQAModel Scale=650M2025.12 | 2.497 | |
| YOCOModel Scale=650M2025.12 | 2.498 | |
| CLAModel Scale=650M, Cache Mem.=1/22025.12 | 2.511 | |
| FusedKV-LiteModel Scale=332M2025.12 | 2.639 | |
| FusedKVModel Scale=332M2025.12 | 2.642 | |
| VanillaModel Scale=332M, Cache Mem.=12025.12 | 2.651 | |
| GQAModel Scale=332M2025.12 | 2.662 | |
| YOCOModel Scale=332M2025.12 | 2.663 | |
| CLAModel Scale=332M, Cache Mem.=1/22025.12 | 2.676 |