Language Modeling on 1 Billion Word Language Modeling Benchmark holdout (test)
28Test Perplexity (10 epochs)MoE-143M
Evaluation results
| Method | Links | ||
|---|---|---|---|
| MoE-143Mops/timestep (millions)=142.7, #Params excluding embed. & softmax (millions)=4371.1, Total #Params (billions)=6, DropProb=0.4, TFLOPS per GPU (observed)=1.562017.01 | 28 | — | |
| MoE-34Mops/timestep (millions)=33.8, #Params excluding embed. & softmax (millions)=4313.9, Total #Params (billions)=6, DropProb=0.3, TFLOPS per GPU (observed)=1.222017.01 | 31.3 | — | |
| MoE-4096-hops/timestep (millions)=8.9, #Params excluding embed. & softmax (millions)=4303.4, Total #Params (billions)=5.1, DropProb=0.2, TFLOPS per GPU (observed)=0.742017.01 | 34.1 | — | |
| MoE-1024-hops/timestep (millions)=8.5, #Params excluding embed. & softmax (millions)=1079, Total #Params (billions)=1.9, DropProb=0.2, TFLOPS per GPU (observed)=0.92017.01 | 34.6 | — | |
| 2xLSTM-8192-1024Source=Jozefowicz et al., 2016, ops/timestep (millions)=151, #Params excluding embed. & softmax (millions)=151, Total #Params (billions)=1.8, DropProb=0.25, TFLOPS per GPU (observed)=1.092017.01 | 34.7 | 30.6 | |
| MoE-256ops/timestep (millions)=8.6, #Params excluding embed. & softmax (millions)=272.9, Total #Params (billions)=1.1, DropProb=0.1, TFLOPS per GPU (observed)=0.812017.01 | 35.7 | — | |
| MoE-256-hops/timestep (millions)=8.4, #Params excluding embed. & softmax (millions)=272.9, Total #Params (billions)=1.1, DropProb=0.1, TFLOPS per GPU (observed)=0.892017.01 | 36 | — | |
| MoE-32ops/timestep (millions)=8.4, #Params excluding embed. & softmax (millions)=37.8, Total #Params (billions)=0.9, DropProb=0.1, TFLOPS per GPU (observed)=0.872017.01 | 39.7 | — | |
| LSTM-2048-512Source=Current Paper Rerun, ops/timestep (millions)=9.4, #Params excluding embed. & softmax (millions)=9.4, Total #Params (billions)=0.8, DropProb=0.1, TFLOPS per GPU (observed)=1.212017.01 | 44.7 | — | |
| LSTM-2048-512Source=Jozefowicz et al., 2016, ops/timestep (millions)=9.4, #Params excluding embed. & softmax (millions)=9.4, Total #Params (billions)=0.8, DropProb=0.1, TFLOPS per GPU (observed)=0.612017.01 | 45 | 43.7 | |
| MoE-4ops/timestep (millions)=8.4, #Params excluding embed. & softmax (millions)=8.4, Total #Params (billions)=0.8, DropProb=0.1, TFLOPS per GPU (observed)=0.522017.01 | 45 | — | |
| MoE-1-Deepops/timestep (millions)=8.4, #Params excluding embed. & softmax (millions)=8.4, Total #Params (billions)=0.8, DropProb=0.1, TFLOPS per GPU (observed)=1.292017.01 | 45.7 | — | |
| 4xLSTM-512ops/timestep (millions)=8.4, #Params excluding embed. & softmax (millions)=8.4, Total #Params (billions)=0.8, DropProb=0.1, TFLOPS per GPU (observed)=1.072017.01 | 46 | — | |
| MoE-1-Wideops/timestep (millions)=8.4, #Params excluding embed. & softmax (millions)=8.4, Total #Params (billions)=0.8, DropProb=0.1, TFLOPS per GPU (observed)=1.292017.01 | 46.1 | — | |
| Kneser-Ney 5-gramops/timestep (millions)=0.00001, Total #Params (billions)=1.82017.01 | — | 67.6 | |
| LSTM-1024-512Source=Jozefowicz et al., 2016, ops/timestep (millions)=4.7, #Params excluding embed. & softmax (millions)=4.7, Total #Params (billions)=0.8, DropProb=0.12017.01 | — | 48.2 | |
| LSTM-512-512Source=Jozefowicz et al., 2016, ops/timestep (millions)=2.4, #Params excluding embed. & softmax (millions)=2.4, Total #Params (billions)=0.8, DropProb=0.12017.01 | — | 54.1 |