Multi-lingual Language Modeling on Multi-lingual language modeling (val)
5.09PerplexityMH-MoE
Evaluation results
| Method | Links | |
|---|---|---|
| MH-MoENumber of Experts=32 Experts2024.04 | 5.09 | |
| X-MoENumber of Experts=32 Experts2024.04 | 6.02 | |
| MH-MoENumber of Experts=8 Experts2024.04 | 6.26 | |
| X-MoENumber of Experts=8 Experts2024.04 | 7.19 | |
| Dense (without Experts)Number of Experts=8 Experts2024.04 | 8.56 | |
| Dense (without Experts)Number of Experts=32 Experts2024.04 | 8.56 |