English-focused language modeling on English-focused language modeling (val)
10.28PerplexityMH-MoE
Evaluation Results
| Method | Links | |
|---|---|---|
| MH-MoENumber of Experts=32 Experts2024.04 | 10.28 | |
| X-MoENumber of Experts=32 Experts2024.04 | 11.96 | |
| MH-MoENumber of Experts=8 Experts2024.04 | 12.72 | |
| X-MoENumber of Experts=8 Experts2024.04 | 14.82 | |
| Dense (without Experts)Number of Experts=8 Experts2024.04 | 16.23 | |
| Dense (without Experts)Number of Experts=32 Experts2024.04 | 16.23 |