Language Modeling on Code, Science, and Fiction domains Equal-Weight Evaluation
2.09EW LossKALAVAI (MoE)
Evaluation Results
| Method | Links | ||||
|---|---|---|---|---|---|
| KALAVAI (MoE)Scale=Pythia-1B, Seeds=32026.03 | 2.09 | 7.49 | 15.5 | 0.01 | |
| Monolithic baselineScale=Pythia-1B, Seeds=12026.03 | 2.097 | — | 15.3 | — | |
| KALAVAI (MoE)Scale=Pythia-6.9B, Seeds=32026.03 | 2.118 | 6.53 | 8.7 | 0.024 | |
| KALAVAI (MoE)Scale=Pythia-410M, Seeds=32026.03 | 2.218 | 7.72 | 16.3 | 0.02 | |
| Monolithic baselineScale=Pythia-410M, Seeds=32026.03 | 2.229 | — | 16 | 0 | |
| Best specialistScale=Pythia-1B, Seeds=12026.03 | 2.259 | — | 8.7 | — | |
| Best specialistScale=Pythia-6.9B, Seeds=12026.03 | 2.266 | — | 2.3 | — | |
| Base modelScale=Pythia-6.9B, Seeds=—2026.03 | 2.32 | — | — | — | |
| Best specialistScale=Pythia-410M, Seeds=32026.03 | 2.404 | — | 9.3 | 0.02 | |
| Base modelScale=Pythia-1B, Seeds=—2026.03 | 2.474 | — | — | — | |
| Weight averagingScale=Pythia-410M, Seeds=32026.03 | 2.486 | 3.4 | 6.2 | 0 | |
| Base modelScale=Pythia-410M, Seeds=—2026.03 | 2.651 | — | — | — |