List your company
ResearchFormerly WizWand

Masked Language Modeling on Wikipedia + BookCorpus (dev)

74.76MLM AccuracyRealFormer
61.042464.603768.16571.7263Dec 21, 2020
Updated 8mo ago

Evaluation results

MethodLinks
RealFormerModel=BERT-xLarge, Pre-training steps=1M, Number of layers=36, Hidden size=1536, Number of parameters=1B2020.1274.76
RealFormerModel=BERT-Large, Pre-training steps=1M, Number of layers=24, Hidden size=1024, Number of parameters=340M2020.1273.94
Post-LNModel=BERT-xLarge, Pre-training steps=1M, Number of layers=36, Hidden size=1536, Number of parameters=1B2020.1273.72
Post-LNModel=BERT-Large, Pre-training steps=1M, Number of layers=24, Hidden size=1024, Number of parameters=340M2020.1273.64
Pre-LNModel=BERT-xLarge, Pre-training steps=1M, Number of layers=36, Hidden size=1536, Number of parameters=1B2020.1273.53
Pre-LNModel=BERT-Large, Pre-training steps=1M, Number of layers=24, Hidden size=1024, Number of parameters=340M2020.1273.21
RealFormerModel=BERT-Base, Pre-training steps=1M, Number of layers=12, Hidden size=768, Number of parameters=110M2020.1270.42
Post-LNModel=BERT-Base, Pre-training steps=1M, Number of layers=12, Hidden size=768, Number of parameters=110M2020.1270.2
Pre-LNModel=BERT-Base, Pre-training steps=1M, Number of layers=12, Hidden size=768, Number of parameters=110M2020.1269.74
RealFormerModel=BERT-Small, Pre-training steps=1M, Number of layers=4, Hidden size=512, Number of parameters=30M2020.1261.7
Pre-LNModel=BERT-Small, Pre-training steps=1M, Number of layers=4, Hidden size=512, Number of parameters=30M2020.1261.67
Post-LNModel=BERT-Small, Pre-training steps=1M, Number of layers=4, Hidden size=512, Number of parameters=30M2020.1261.57