List your company

Language Modeling on Code, Science, and Fiction domains Equal-Weight Evaluation

2.09EW LossKALAVAI (MoE)
2.067562.219032.37052.52197Mar 24, 2026

Evaluation results

MethodLinks
KALAVAI (MoE)Scale=Pythia-1B, Seeds=32026.032.097.4915.50.01
Monolithic baselineScale=Pythia-1B, Seeds=12026.032.097—15.3—
KALAVAI (MoE)Scale=Pythia-6.9B, Seeds=32026.032.1186.538.70.024
KALAVAI (MoE)Scale=Pythia-410M, Seeds=32026.032.2187.7216.30.02
Monolithic baselineScale=Pythia-410M, Seeds=32026.032.229—160
Best specialistScale=Pythia-1B, Seeds=12026.032.259—8.7—
Best specialistScale=Pythia-6.9B, Seeds=12026.032.266—2.3—
Base modelScale=Pythia-6.9B, Seeds=—2026.032.32———
Best specialistScale=Pythia-410M, Seeds=32026.032.404—9.30.02
Base modelScale=Pythia-1B, Seeds=—2026.032.474———
Weight averagingScale=Pythia-410M, Seeds=32026.032.4863.46.20
Base modelScale=Pythia-410M, Seeds=—2026.032.651———