Language Modeling on WikiText-103 clean (test)
23.32PerplexityTransformer-XL
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Transformer-XLModel size (Parameters)=Base (151M), Attention=QUEST2026.03 | 23.32 | — | |
| Transformer-XLModel size (Parameters)=Base (151M), Attention=Standard2026.03 | 23.592 | — | |
| TransformerModel size (Parameters)=Medium (90M), Attention=QUEST2026.03 | 28.478 | — | |
| TransformerModel size (Parameters)=Medium (90M), Attention=QKNorm-DS2026.03 | 28.624 | — | |
| TransformerModel size (Parameters)=Medium (90M), Attention=QNorm2026.03 | 28.688 | — | |
| TransformerModel size (Parameters)=Medium (90M), Attention=Standard2026.03 | 28.851 | — | |
| SSR-L w/ noiseSetting=Momentum setting, Training Time Overhead=1.58%2025.11 | 33.223 | -0.489 | |
| SSR-LSetting=Momentum setting, Training Time Overhead=0.36%2025.11 | 33.305 | -0.407 | |
| SSR-S w/ noiseSetting=Momentum setting, Training Time Overhead=1.76%2025.11 | 33.338 | -0.374 | |
| SSR-SSetting=Momentum setting, Training Time Overhead=0.61%2025.11 | 33.559 | -0.153 | |
| Vanilla SMoESetting=Momentum setting, Training Time Overhead=0.00%2025.11 | 33.712 | — | |
| TransformerModel size (Parameters)=Small (44M), Attention=QUEST2026.03 | 33.966 | — | |
| TransformerModel size (Parameters)=Small (44M), Attention=Standard2026.03 | 34.076 | — | |
| SMoE w/ noiseSetting=Momentum setting, Training Time Overhead=6.75%2025.11 | 34.077 | 0.365 | |
| SMoE w/ z-lossSetting=Momentum setting, Training Time Overhead=4.44%2025.11 | 34.235 | 0.523 | |
| Sinkhorn-based SMoESetting=Momentum setting, Training Time Overhead=67.52%2025.11 | 34.305 | 0.593 | |
| SSR-L w/ noiseSetting=Conventional setting, Training Time Overhead=0.62%2025.11 | 34.367 | -1.183 | |
| Switch-ACMoEBackbone=Switch2025.02 | 34.42 | — | |
| SMoE w/ lb lossSetting=Momentum setting, Training Time Overhead=2.49%2025.11 | 34.49 | 0.778 | |
| SSR-S w/ noiseSetting=Conventional setting, Training Time Overhead=0.65%2025.11 | 34.557 | -0.993 | |
| SSR-LSetting=Conventional setting, Training Time Overhead=0.33%2025.11 | 34.573 | -0.977 | |
| SSR-SSetting=Conventional setting, Training Time Overhead=0.37%2025.11 | 34.744 | -0.806 | |
| SMoE w/ lb lossSetting=Conventional setting, Training Time Overhead=2.04 %2025.11 | 34.952 | -0.598 | |
| SMoE w/ noiseSetting=Conventional setting, Training Time Overhead=2.50%2025.11 | 35.072 | -0.478 | |
| SMoE w/ z-lossSetting=Conventional setting, Training Time Overhead=2.12%2025.11 | 35.091 | -0.459 | |
| Sinkhorn-based SMoESetting=Conventional setting, Training Time Overhead=72.47%2025.11 | 35.316 | -0.234 | |
| Switch TransformerBackbone=Switch2025.02 | 35.48 | — | |
| Vanilla SMoESetting=Conventional setting, Training Time Overhead=0.00%2025.11 | 35.55 | — | |
| GLaM-ACMoEBackbone=GLaM2025.02 | 36.26 | — | |
| GLaMBackbone=GLaM2025.02 | 38.27 | — |