Language Modeling on Private-domain fusion set medical, legal, and patent
2.418EW LossKALAVAI (MoE)
Evaluation Results
| Method | Links | ||||
|---|---|---|---|---|---|
| KALAVAI (MoE)Scale=Pythia-410M, Seeds=32026.03 | 2.418 | 0.1017 | 0.0178 | 0.0015 | |
| Monolithic (6k)Scale=Pythia-410M, Seeds=12026.03 | 2.462 | — | — | — | |
| Best specialistScale=Pythia-410M, Seeds=32026.03 | 2.694 | — | — | — | |
| Weight avg.Scale=Pythia-410M2026.03 | 2.709 | — | — | — | |
| Base modelScale=Pythia-410M2026.03 | 2.954 | — | — | — |