Average Accuracy on 9 Downstream Benchmarks Average
0.4563Average AccuracyPowerMoE-3B
Evaluation results
| Method | Links | |
|---|---|---|
| PowerMoE-3BTotal Parameters=3B, Active Parameters=0.8B, Training Tokens=3T2026.06 | 0.4563 | |
| OLMoE-1B-7BTotal Parameters=7B, Active Parameters=1B, Training Tokens=200B2026.06 | 0.3738 | |
| LoopMoETotal Parameters=3B, Active Parameters=0.8B‡, Training Tokens=200B2026.06 | 0.3589 | |
| Vanilla MoETotal Parameters=3B, Active Parameters=0.8B, Training Tokens=200B2026.06 | 0.3484 | |
| OLMo2-1BTotal Parameters=1.0B, Active Parameters=1.0B, Training Tokens=200B2026.06 | 0.3212 | |
| Pythia-1.4BTotal Parameters=1.4B, Active Parameters=1.4B, Training Tokens=200B2026.06 | 0.313 |