Language Modeling on WikiText-103 Attacked (test)
41.351Perplexity (PPL)SSR-L
Evaluation Results
| Method | Links | |
|---|---|---|
| SSR-LSetting=Momentum setting, Training Time Overhead=0.36%2025.11 | 41.351 | |
| SSR-L w/ noiseSetting=Momentum setting, Training Time Overhead=1.58%2025.11 | 41.627 | |
| SSR-S w/ noiseSetting=Momentum setting, Training Time Overhead=1.76%2025.11 | 41.639 | |
| Vanilla SMoESetting=Momentum setting, Training Time Overhead=0.00%2025.11 | 41.756 | |
| SSR-SSetting=Momentum setting, Training Time Overhead=0.61%2025.11 | 42.104 | |
| SMoE w/ noiseSetting=Momentum setting, Training Time Overhead=6.75%2025.11 | 42.204 | |
| SMoE w/ z-lossSetting=Momentum setting, Training Time Overhead=4.44%2025.11 | 42.614 | |
| Sinkhorn-based SMoESetting=Momentum setting, Training Time Overhead=67.52%2025.11 | 42.803 | |
| SmoothSMoETotal Experts=16, Routing=top-22026.06 | 42.85 | |
| SSR-S w/ noiseSetting=Conventional setting, Training Time Overhead=0.65%2025.11 | 42.941 | |
| SMoE w/ lb lossSetting=Momentum setting, Training Time Overhead=2.49%2025.11 | 42.998 | |
| SSR-L w/ noiseSetting=Conventional setting, Training Time Overhead=0.62%2025.11 | 43.159 | |
| SSR-SSetting=Conventional setting, Training Time Overhead=0.37%2025.11 | 43.283 | |
| SSR-LSetting=Conventional setting, Training Time Overhead=0.33%2025.11 | 43.318 | |
| SMoE w/ noiseSetting=Conventional setting, Training Time Overhead=2.50%2025.11 | 43.692 | |
| SMoE w/ lb lossSetting=Conventional setting, Training Time Overhead=2.04 %2025.11 | 43.758 | |
| ReMoETotal Experts=16, Routing=top-22026.06 | 44 | |
| SMoETotal Experts=16, Routing=top-22026.06 | 44.18 | |
| Vanilla SMoESetting=Conventional setting, Training Time Overhead=0.00%2025.11 | 44.19 | |
| Sinkhorn-based SMoESetting=Conventional setting, Training Time Overhead=72.47%2025.11 | 44.351 | |
| SMoE w/ z-lossSetting=Conventional setting, Training Time Overhead=2.12%2025.11 | 44.406 |