Masked Language Modeling on Wikipedia + BookCorpus (dev)
74.76MLM AccuracyRealFormer
Evaluation Results
| Method | Links | |
|---|---|---|
| RealFormerModel=BERT-xLarge, Pre-training steps=1M, Number of layers=36, Hidden size=1536, Number of parameters=1B2020.12 | 74.76 | |
| RealFormerModel=BERT-Large, Pre-training steps=1M, Number of layers=24, Hidden size=1024, Number of parameters=340M2020.12 | 73.94 | |
| Post-LNModel=BERT-xLarge, Pre-training steps=1M, Number of layers=36, Hidden size=1536, Number of parameters=1B2020.12 | 73.72 | |
| Post-LNModel=BERT-Large, Pre-training steps=1M, Number of layers=24, Hidden size=1024, Number of parameters=340M2020.12 | 73.64 | |
| Pre-LNModel=BERT-xLarge, Pre-training steps=1M, Number of layers=36, Hidden size=1536, Number of parameters=1B2020.12 | 73.53 | |
| Pre-LNModel=BERT-Large, Pre-training steps=1M, Number of layers=24, Hidden size=1024, Number of parameters=340M2020.12 | 73.21 | |
| RealFormerModel=BERT-Base, Pre-training steps=1M, Number of layers=12, Hidden size=768, Number of parameters=110M2020.12 | 70.42 | |
| Post-LNModel=BERT-Base, Pre-training steps=1M, Number of layers=12, Hidden size=768, Number of parameters=110M2020.12 | 70.2 | |
| Pre-LNModel=BERT-Base, Pre-training steps=1M, Number of layers=12, Hidden size=768, Number of parameters=110M2020.12 | 69.74 | |
| RealFormerModel=BERT-Small, Pre-training steps=1M, Number of layers=4, Hidden size=512, Number of parameters=30M2020.12 | 61.7 | |
| Pre-LNModel=BERT-Small, Pre-training steps=1M, Number of layers=4, Hidden size=512, Number of parameters=30M2020.12 | 61.67 | |
| Post-LNModel=BERT-Small, Pre-training steps=1M, Number of layers=4, Hidden size=512, Number of parameters=30M2020.12 | 61.57 |