Language Modeling on WikiText-103 Attacked (val)
39.677PerplexitySSR-S w/ noise
Evaluation Results
| Method | Links | |
|---|---|---|
| SSR-S w/ noiseSetting=Momentum setting, Training Time Overhead=1.76%2025.11 | 39.677 | |
| SSR-LSetting=Momentum setting, Training Time Overhead=0.36%2025.11 | 39.69 | |
| Vanilla SMoESetting=Momentum setting, Training Time Overhead=0.00%2025.11 | 39.721 | |
| SSR-L w/ noiseSetting=Momentum setting, Training Time Overhead=1.58%2025.11 | 39.847 | |
| SSR-SSetting=Momentum setting, Training Time Overhead=0.61%2025.11 | 40.259 | |
| SMoE w/ noiseSetting=Momentum setting, Training Time Overhead=6.75%2025.11 | 40.299 | |
| SMoE w/ z-lossSetting=Momentum setting, Training Time Overhead=4.44%2025.11 | 40.665 | |
| Sinkhorn-based SMoESetting=Momentum setting, Training Time Overhead=67.52%2025.11 | 40.81 | |
| SMoE w/ lb lossSetting=Momentum setting, Training Time Overhead=2.49%2025.11 | 40.811 | |
| SmoothSMoETotal Experts=16, Routing=top-22026.06 | 40.99 | |
| SSR-SSetting=Conventional setting, Training Time Overhead=0.37%2025.11 | 41.012 | |
| SSR-S w/ noiseSetting=Conventional setting, Training Time Overhead=0.65%2025.11 | 41.037 | |
| SSR-L w/ noiseSetting=Conventional setting, Training Time Overhead=0.62%2025.11 | 41.164 | |
| SSR-LSetting=Conventional setting, Training Time Overhead=0.33%2025.11 | 41.23 | |
| SMoE w/ noiseSetting=Conventional setting, Training Time Overhead=2.50%2025.11 | 41.432 | |
| SMoE w/ lb lossSetting=Conventional setting, Training Time Overhead=2.04 %2025.11 | 41.763 | |
| Sinkhorn-based SMoESetting=Conventional setting, Training Time Overhead=72.47%2025.11 | 42.056 | |
| ReMoETotal Experts=16, Routing=top-22026.06 | 42.19 | |
| SMoETotal Experts=16, Routing=top-22026.06 | 42.21 | |
| Vanilla SMoESetting=Conventional setting, Training Time Overhead=0.00%2025.11 | 42.24 | |
| SMoE w/ z-lossSetting=Conventional setting, Training Time Overhead=2.12%2025.11 | 42.298 |