Language Modeling on WikiText-103 Clean (val)
31.82PerplexitySSR-S w/ noise
Evaluation Results
| Method | Links | |
|---|---|---|
| SSR-S w/ noiseSetting=Momentum setting, Training Time Overhead=1.76%2025.11 | 31.82 | |
| SSR-LSetting=Momentum setting, Training Time Overhead=0.36%2025.11 | 31.834 | |
| Vanilla SMoESetting=Momentum setting, Training Time Overhead=0.00%2025.11 | 31.861 | |
| SSR-SSetting=Momentum setting, Training Time Overhead=0.61%2025.11 | 31.888 | |
| SSR-L w/ noiseSetting=Momentum setting, Training Time Overhead=1.58%2025.11 | 31.911 | |
| SMoE w/ noiseSetting=Momentum setting, Training Time Overhead=6.75%2025.11 | 32.49 | |
| SMoE w/ z-lossSetting=Momentum setting, Training Time Overhead=4.44%2025.11 | 32.542 | |
| SMoE w/ lb lossSetting=Momentum setting, Training Time Overhead=2.49%2025.11 | 32.561 | |
| Sinkhorn-based SMoESetting=Momentum setting, Training Time Overhead=67.52%2025.11 | 32.576 | |
| SSR-SSetting=Conventional setting, Training Time Overhead=0.37%2025.11 | 32.61 | |
| SSR-L w/ noiseSetting=Conventional setting, Training Time Overhead=0.62%2025.11 | 32.767 | |
| SSR-LSetting=Conventional setting, Training Time Overhead=0.33%2025.11 | 32.809 | |
| SSR-S w/ noiseSetting=Conventional setting, Training Time Overhead=0.65%2025.11 | 32.881 | |
| SMoE w/ noiseSetting=Conventional setting, Training Time Overhead=2.50%2025.11 | 33.196 | |
| SMoE w/ z-lossSetting=Conventional setting, Training Time Overhead=2.12%2025.11 | 33.242 | |
| SMoE w/ lb lossSetting=Conventional setting, Training Time Overhead=2.04 %2025.11 | 33.248 | |
| Sinkhorn-based SMoESetting=Conventional setting, Training Time Overhead=72.47%2025.11 | 33.301 | |
| Vanilla SMoESetting=Conventional setting, Training Time Overhead=0.00%2025.11 | 33.76 |