Language Modeling on Cosmopedia v2
6.49Final Perplexity (PPL)MoL K=5 top-3
Evaluation results
| Method | Links | ||
|---|---|---|---|
| MoL K=5 top-3Params=104.0M2026.05 | 6.49 | -0.16 | |
| Sparse MoL 1+2of5Params=106.0M2026.05 | 6.52 | -0.13 | |
| Dense DeltaNetParams=103.4M2026.05 | 6.64 | -0.01 | |
| Dense softmaxParams=103.4M2026.05 | 6.65 | — | |
| MoE-FFN 3of5Params=~106M2026.05 | 6.89 | 0.24 |