Language Modeling on Wikipedia languagemodel_wiki_noref_v128k_11k
6.01Subword PerplexityMesh-TensorFlow (Transformer-Decoder)
Evaluation Results
| Method | Links | |
|---|---|---|
| Mesh-TensorFlow (Transformer-Decoder)d_ff=262144, heads=256, Parameters (Billions)=4.92018.11 | 6.01 | |
| Mesh-TensorFlow (Transformer-Decoder)d_ff=131072, heads=128, Parameters (Billions)=2.482018.11 | 6.24 | |
| Mesh-TensorFlow (Transformer-Decoder)d_ff=65536, heads=64, Parameters (Billions)=1.282018.11 | 6.55 | |
| Mesh-TensorFlow (Transformer-Decoder)d_ff=32768, heads=32, Parameters (Billions)=0.672018.11 | 6.99 | |
| Mesh-TensorFlow (Transformer-Decoder)d_ff=16384, heads=16, Parameters (Billions)=0.372018.11 | 7.44 | |
| Mesh-TensorFlow (Transformer-Decoder)d_ff=8192, heads=8, Parameters (Billions)=0.222018.11 | 8.03 | |
| Mesh-TensorFlow (Transformer-Decoder)d_ff=4096, heads=4, Parameters (Billions)=0.142018.11 | 8.74 |