Masked Language Modeling on Books, CC-News, Stories, Wikipedia (held-out set)
1.274BPCBIGBIRD-ETC
Evaluation Results
| Method | Links | |
|---|---|---|
| BIGBIRD-ETCSize=Large, Sequence Length=40962020.07 | 1.274 | |
| LongformerSize=Large, Sequence Length=40962020.07 | 1.358 | |
| BIGBIRD-ITCSize=Large, Sequence Length=40962020.07 | 1.456 | |
| ROBERTaSize=Large, Sequence Length=5122020.07 | 1.496 | |
| BIGBIRD-ETCSize=Base, Sequence Length=40962020.07 | 1.611 | |
| BIGBIRD-ITCSize=Base, Sequence Length=40962020.07 | 1.678 | |
| LongformerSize=Base, Sequence Length=40962020.07 | 1.705 | |
| ROBERTaSize=Base, Sequence Length=5122020.07 | 1.846 |