List your company
ResearchFormerly WizWand

Language Modeling on Billion-Word Benchmark (dev)

23.5Word-PerplexityMesh-TensorFlow (Transformer-Decoder)
23.0426.14529.2532.355Nov 5, 2018
Updated 8mo ago

Evaluation results

MethodLinks
Mesh-TensorFlow (Transformer-Decoder)d_ff=262144, heads=256, Parameters (Billions)=4.9, Logit Scaling=0.92018.1123.5
Best Ensemble (different methods)Parameters (Billions)=> 1002018.1123.7
Mesh-TensorFlow (Transformer-Decoder)d_ff=262144, heads=256, Parameters (Billions)=4.92018.1124
Mesh-TensorFlow (Transformer-Decoder)d_ff=131072, heads=128, Parameters (Billions)=2.482018.1124.1
Mesh-TensorFlow (Transformer-Decoder)d_ff=65536, heads=64, Parameters (Billions)=1.282018.1125.1
Best DNN Ensemble2018.1126.1
Mesh-TensorFlow (Transformer-Decoder)d_ff=32768, heads=32, Parameters (Billions)=0.672018.1126.8
Prev Best DNNParameters (Billions)=6.52018.1128
Mesh-TensorFlow (Transformer-Decoder)d_ff=16384, heads=16, Parameters (Billions)=0.372018.1128.9
Mesh-TensorFlow (Transformer-Decoder)d_ff=8192, heads=8, Parameters (Billions)=0.222018.1131.7
Mesh-TensorFlow (Transformer-Decoder)d_ff=4096, heads=4, Parameters (Billions)=0.142018.1135