Language Modeling on Billion-Word Benchmark (dev)
23.5Word-PerplexityMesh-TensorFlow (Transformer-Decoder)
Evaluation Results
| Method | Links | |
|---|---|---|
| Mesh-TensorFlow (Transformer-Decoder)d_ff=262144, heads=256, Parameters (Billions)=4.9, Logit Scaling=0.92018.11 | 23.5 | |
| Best Ensemble (different methods)Parameters (Billions)=> 1002018.11 | 23.7 | |
| Mesh-TensorFlow (Transformer-Decoder)d_ff=262144, heads=256, Parameters (Billions)=4.92018.11 | 24 | |
| Mesh-TensorFlow (Transformer-Decoder)d_ff=131072, heads=128, Parameters (Billions)=2.482018.11 | 24.1 | |
| Mesh-TensorFlow (Transformer-Decoder)d_ff=65536, heads=64, Parameters (Billions)=1.282018.11 | 25.1 | |
| Best DNN Ensemble2018.11 | 26.1 | |
| Mesh-TensorFlow (Transformer-Decoder)d_ff=32768, heads=32, Parameters (Billions)=0.672018.11 | 26.8 | |
| Prev Best DNNParameters (Billions)=6.52018.11 | 28 | |
| Mesh-TensorFlow (Transformer-Decoder)d_ff=16384, heads=16, Parameters (Billions)=0.372018.11 | 28.9 | |
| Mesh-TensorFlow (Transformer-Decoder)d_ff=8192, heads=8, Parameters (Billions)=0.222018.11 | 31.7 | |
| Mesh-TensorFlow (Transformer-Decoder)d_ff=4096, heads=4, Parameters (Billions)=0.142018.11 | 35 |