List your company
ResearchFormerly WizWand

Language Modeling on WikiText-103 Clean (val)

31.82PerplexitySSR-S w/ noise
31.742432.266232.7933.3138Nov 12, 2025
Updated 5mo ago

Evaluation results

MethodLinks
SSR-S w/ noiseSetting=Momentum setting, Training Time Overhead=1.76%2025.1131.82
SSR-LSetting=Momentum setting, Training Time Overhead=0.36%2025.1131.834
Vanilla SMoESetting=Momentum setting, Training Time Overhead=0.00%2025.1131.861
SSR-SSetting=Momentum setting, Training Time Overhead=0.61%2025.1131.888
SSR-L w/ noiseSetting=Momentum setting, Training Time Overhead=1.58%2025.1131.911
SMoE w/ noiseSetting=Momentum setting, Training Time Overhead=6.75%2025.1132.49
SMoE w/ z-lossSetting=Momentum setting, Training Time Overhead=4.44%2025.1132.542
SMoE w/ lb lossSetting=Momentum setting, Training Time Overhead=2.49%2025.1132.561
Sinkhorn-based SMoESetting=Momentum setting, Training Time Overhead=67.52%2025.1132.576
SSR-SSetting=Conventional setting, Training Time Overhead=0.37%2025.1132.61
SSR-L w/ noiseSetting=Conventional setting, Training Time Overhead=0.62%2025.1132.767
SSR-LSetting=Conventional setting, Training Time Overhead=0.33%2025.1132.809
SSR-S w/ noiseSetting=Conventional setting, Training Time Overhead=0.65%2025.1132.881
SMoE w/ noiseSetting=Conventional setting, Training Time Overhead=2.50%2025.1133.196
SMoE w/ z-lossSetting=Conventional setting, Training Time Overhead=2.12%2025.1133.242
SMoE w/ lb lossSetting=Conventional setting, Training Time Overhead=2.04 %2025.1133.248
Sinkhorn-based SMoESetting=Conventional setting, Training Time Overhead=72.47%2025.1133.301
Vanilla SMoESetting=Conventional setting, Training Time Overhead=0.00%2025.1133.76