Masked Language Modeling on BERT Pretraining Corpus
2.89PerplexitygMLP_xlarge
Evaluation results
| Method | Links | |
|---|---|---|
| gMLP_xlargeParams (M)=9412021.05 | 2.89 | |
| aMLP_largeAttn Size=128, Params (M)=3162021.05 | 3.19 | |
| gMLP_largeParams (M)=3652021.05 | 3.32 | |
| BERT_large (ours)Attn Size=1024 (64 x 16), Params (M)=3362021.05 | 3.35 | |
| aMLP_baseAttn Size=64, Params (M)=1092021.05 | 3.95 | |
| BERT_base (ours)Attn Size=768 (64 x 12), Params (M)=1102021.05 | 4.17 | |
| gMLP_baseParams (M)=1302021.05 | 4.28 |