Language Modeling on LAMBADA 5-shot (Accuracy)
63.4Accuracy (LAMBADA 5-shot)MoE 128e8a1s
Evaluation Results
| Method | Links | |
|---|---|---|
| MoE 128e8a1sModel=MoE 128e8a1s, Training Steps=100k2026.05 | 63.4 | |
| MoE 128e8a4g1sModel=MoE 128e8a4g1s, Training Steps=100k2026.05 | 61.9 | |
| DenseModel=Dense, Training Steps=100k2026.05 | 50.5 |