List your company
ResearchFormerly WizWand

Language Modeling on WikiText-103 Attacked (val)

39.677PerplexitySSR-S w/ noise
39.5721640.2798340.987541.69517Nov 12, 2025Dec 18, 2025Jan 23, 2026Feb 28, 2026Apr 5, 2026May 11, 2026Jun 17, 2026
Updated 4mo ago

Evaluation results

MethodLinks
SSR-S w/ noiseSetting=Momentum setting, Training Time Overhead=1.76%2025.1139.677
SSR-LSetting=Momentum setting, Training Time Overhead=0.36%2025.1139.69
Vanilla SMoESetting=Momentum setting, Training Time Overhead=0.00%2025.1139.721
SSR-L w/ noiseSetting=Momentum setting, Training Time Overhead=1.58%2025.1139.847
SSR-SSetting=Momentum setting, Training Time Overhead=0.61%2025.1140.259
SMoE w/ noiseSetting=Momentum setting, Training Time Overhead=6.75%2025.1140.299
SMoE w/ z-lossSetting=Momentum setting, Training Time Overhead=4.44%2025.1140.665
Sinkhorn-based SMoESetting=Momentum setting, Training Time Overhead=67.52%2025.1140.81
SMoE w/ lb lossSetting=Momentum setting, Training Time Overhead=2.49%2025.1140.811
SmoothSMoETotal Experts=16, Routing=top-22026.0640.99
SSR-SSetting=Conventional setting, Training Time Overhead=0.37%2025.1141.012
SSR-S w/ noiseSetting=Conventional setting, Training Time Overhead=0.65%2025.1141.037
SSR-L w/ noiseSetting=Conventional setting, Training Time Overhead=0.62%2025.1141.164
SSR-LSetting=Conventional setting, Training Time Overhead=0.33%2025.1141.23
SMoE w/ noiseSetting=Conventional setting, Training Time Overhead=2.50%2025.1141.432
SMoE w/ lb lossSetting=Conventional setting, Training Time Overhead=2.04 %2025.1141.763
Sinkhorn-based SMoESetting=Conventional setting, Training Time Overhead=72.47%2025.1142.056
ReMoETotal Experts=16, Routing=top-22026.0642.19
SMoETotal Experts=16, Routing=top-22026.0642.21
Vanilla SMoESetting=Conventional setting, Training Time Overhead=0.00%2025.1142.24
SMoE w/ z-lossSetting=Conventional setting, Training Time Overhead=2.12%2025.1142.298