List your company
ResearchFormerly WizWand

Masked Language Modeling on Books, CC-News, Stories, Wikipedia (held-out set)

1.274BPCBIGBIRD-ETC
1.251121.405561.561.71444Jul 28, 2020
Updated 8mo ago

Evaluation results

MethodLinks
BIGBIRD-ETCSize=Large, Sequence Length=40962020.071.274
LongformerSize=Large, Sequence Length=40962020.071.358
BIGBIRD-ITCSize=Large, Sequence Length=40962020.071.456
ROBERTaSize=Large, Sequence Length=5122020.071.496
BIGBIRD-ETCSize=Base, Sequence Length=40962020.071.611
BIGBIRD-ITCSize=Base, Sequence Length=40962020.071.678
LongformerSize=Base, Sequence Length=40962020.071.705
ROBERTaSize=Base, Sequence Length=5122020.071.846