Language Modeling on WikiText-103 Medium (test)
35.37Perplexity (PPL)NAMEx-Full-Mom
Evaluation Results
| Method | Links | |
|---|---|---|
| NAMEx-Full-MomParams=70M / 216M, Momentum=true2025.10 | 35.37 | |
| SMoE (Top-2)Params=70M / 216M2025.10 | 35.55 | |
| NAMEx-MomParams=70M / 216M, Momentum=true2025.10 | 35.86 | |
| EP-CAMEx-MomParams=70M / 216M, Momentum=true2025.10 | 36.16 | |
| NAMEx-FullParams=70M / 216M, Momentum=false2025.10 | 36.21 | |
| NAMExParams=70M / 216M, Momentum=false2025.10 | 36.4 | |
| CAMExParams=70M / 216M2025.10 | 36.53 | |
| EP-CAMExParams=70M / 216M, Momentum=false2025.10 | 36.55 | |
| SMEARParams=70M / 216M2025.10 | 37.42 | |
| SMoE (Top-1)Params=70M / 216M2025.10 | 40.51 |