Language Modeling on FineWeb-Edu 20B tokens (val)
15.03Final PPLDense Softmax 1.3B (Exp 085)
Evaluation results
| Method | Links | |
|---|---|---|
| Dense Softmax 1.3B (Exp 085)Total Parameters=1.31B, Active Parameters=1.31B, dff=8192, Wall time=53.3h2026.05 | 15.03 | |
| MoL Hybrid 1+3of15 (Exp 087)Total Parameters=2.08B, Active Parameters=0.61B, dff=2048†, Wall time=102.0h2026.05 | 18.04 | |
| Dense Softmax 0.7B (Exp 089)Total Parameters=0.71B, Active Parameters=0.71B, dff=2048, Wall time=44.5h2026.05 | 18.53 |