Language Modeling on Enwik-8 (test) comparing against vanilla SMoE
1.128BPCSSR-S w/ noise
Evaluation results
| Method | Links | ||
|---|---|---|---|
| SSR-S w/ noisenoise=true2025.11 | 1.128 | -0.01 | |
| Sinkhorn-based SMoE2025.11 | 1.133 | -0.005 | |
| SSR-L w/ noisenoise=true2025.11 | 1.133 | -0.005 | |
| SMoE w/ aux. lossauxiliary loss=true2025.11 | 1.136 | -0.002 | |
| Vanilla SMoE2025.11 | 1.138 | — | |
| SMoE w/ noisenoise=true2025.11 | 1.147 | 0.009 |