Multi-task Language Understanding on MMLU (Overall, Stem)
61.08Overall AccuracyFrozen Router
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Frozen RouterBackbone=Qwen1.5-MoE-A2.7B, Top-k=42026.06 | 61.08 | 53.19 | |
| ProbMoEBackbone=Qwen1.5-MoE-A2.7B, Top-k=42026.06 | 61.05 | 53.82 | |
| ConventionalBackbone=Qwen1.5-MoE-A2.7B, Top-k=42026.06 | 61.03 | 53.16 | |
| DenseMixerBackbone=Qwen1.5-MoE-A2.7B, Top-k=42026.06 | 61.03 | 52.87 | |
| Base ModelBackbone=Qwen1.5-MoE-A2.7B, Top-k=42026.06 | 60.85 | 52.27 | |
| ConventionalBackbone=OLMoE-1B-7B, Top-k=82026.06 | 54.04 | 46.46 | |
| Frozen RouterBackbone=OLMoE-1B-7B, Top-k=82026.06 | 53.97 | 46.18 | |
| DenseMixerBackbone=OLMoE-1B-7B, Top-k=82026.06 | 53.95 | 46.18 | |
| ProbMoEBackbone=OLMoE-1B-7B, Top-k=82026.06 | 53.69 | 45.54 | |
| Base ModelBackbone=OLMoE-1B-7B, Top-k=82026.06 | 53.4 | 44.91 |