Masked multi-modal modeling on Masked multi-modal modeling (val)
10.87PerplexityMH-MoE
Evaluation Results
| Method | Links | |
|---|---|---|
| MH-MoENumber of Experts=32 Experts2024.04 | 10.87 | |
| X-MoENumber of Experts=32 Experts2024.04 | 12.68 | |
| MH-MoENumber of Experts=8 Experts2024.04 | 14.73 | |
| X-MoENumber of Experts=8 Experts2024.04 | 16.34 | |
| Dense (without Experts)Number of Experts=8 Experts2024.04 | 17.95 | |
| Dense (without Experts)Number of Experts=32 Experts2024.04 | 17.95 |