Language Modeling on WikiText-103 Medium (val)
33.76Perplexity (PPL)SMoE (Top-2)
Evaluation Results
| Method | Links | |
|---|---|---|
| SMoE (Top-2)Params=70M / 216M2025.10 | 33.76 | |
| NAMEx-Full-MomParams=70M / 216M, Momentum=true2025.10 | 34.25 | |
| NAMEx-MomParams=70M / 216M, Momentum=true2025.10 | 34.89 | |
| NAMEx-FullParams=70M / 216M, Momentum=false2025.10 | 34.92 | |
| EP-CAMEx-MomParams=70M / 216M, Momentum=true2025.10 | 35.09 | |
| NAMExParams=70M / 216M, Momentum=false2025.10 | 35.14 | |
| CAMExParams=70M / 216M2025.10 | 35.69 | |
| EP-CAMExParams=70M / 216M, Momentum=false2025.10 | 35.78 | |
| SMEARParams=70M / 216M2025.10 | 36.15 | |
| SMoE (Top-1)Params=70M / 216M2025.10 | 38.6 |