Language Modeling on OpenWebText (held-out set)
11.5PPLGrMoE + Amort.
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| GrMoE + Amort.Scale=2.7B, N=322026.02 | 11.5 | 0.092 | 0 | 1.38 | |
| Soft MoEScale=2.7B, N=322026.02 | 11.8 | 0.148 | 20 | 1.48 | |
| Softmax Top-2Scale=2.7B, N=322026.02 | 12.1 | 0.342 | 65 | 1.04 | |
| GrMoE + Amort.Scale=1.3B, N=162026.02 | 13.8 | 0.081 | 0 | 1.42 | |
| Soft MoEScale=1.3B, N=162026.02 | 14 | 0.128 | 15 | 1.51 | |
| Softmax Top-2Scale=1.3B, N=162026.02 | 14.2 | 0.298 | 55 | 1.08 | |
| Expert ChoiceScale=1.3B, N=162026.02 | 14.4 | 0.072 | 0 | 1.85 | |
| SwitchScale=1.3B, N=162026.02 | 14.6 | 0.185 | 30 | 0.82 | |
| GrMoE + Amort.Scale=350M, N=82026.02 | 18.1 | 0.074 | 0 | 1.29 | |
| GrMoEScale=350M, N=82026.02 | 18.3 | 0.089 | 0 | 1.34 | |
| Soft MoEScale=350M, N=82026.02 | 18.4 | 0.112 | 10 | 1.42 | |
| vMF-GateScale=350M, N=82026.02 | 18.5 | 0.198 | 20 | 1.18 | |
| Softmax Top-2Scale=350M, N=82026.02 | 18.7 | 0.241 | 40 | 1.12 | |
| Expert ChoiceScale=350M, N=82026.02 | 18.9 | 0.068 | 0 | 1.71 | |
| SwitchScale=350M, N=82026.02 | 19.1 | 0.152 | 20 | 0.89 | |
| HashScale=350M, N=82026.02 | 20.3 | 0.038 | 0 | 2.05 |