# Masked Language Modeling on Books, CC-News, Stories, Wikipedia (held-out set)

Source: [Masked Language Modeling on Books, CC-News, Stories, Wikipedia (held-out set)](<https://www.sota2.com/research/sota/masked-language-modeling-on-books-cc-news-stories-wikipedia-held-out-set>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-02-26T05:37:23.864Z

Evaluation of Masked Language Modeling performance on the held-out test set combining data from Books, CC-News, Stories, and Wikipedia corpora, typically measured using Bits Per Character (BPC).

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

| Scope | Recorded value |
| --- | --- |
| Task | Masked Language Modeling |
| Dataset (including any stated split) | Books, CC-News, Stories, Wikipedia (held-out set) |
| Metric | BPC |
| Unit | SCALAR |
| Direction | Lower is better |
| Results loaded | 8 of 8 |
| Coverage | All results currently returned for this benchmark |

Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol.

## Metrics

- [BPC](<https://www.sota2.com/research/sota/masked-language-modeling-on-books-cc-news-stories-wikipedia-held-out-set.md?metric=BPC>): Lower is better; unit: SCALAR.

## Recorded results

| Method | BPC | Result date | Report paper |
| --- | --- | --- | --- |
| BIGBIRD-ETC | 1.274 | Not recorded | 68fa690b1e209131cea1faf6 |
| Longformer | 1.358 | Not recorded | 68fa690b1e209131cea1faf6 |
| BIGBIRD-ITC | 1.456 | Not recorded | 68fa690b1e209131cea1faf6 |
| ROBERTa | 1.496 | Not recorded | 68fa690b1e209131cea1faf6 |
| BIGBIRD-ETC | 1.611 | Not recorded | 68fa690b1e209131cea1faf6 |
| BIGBIRD-ITC | 1.678 | Not recorded | 68fa690b1e209131cea1faf6 |
| Longformer | 1.705 | Not recorded | 68fa690b1e209131cea1faf6 |
| ROBERTa | 1.846 | Not recorded | 68fa690b1e209131cea1faf6 |

### BIGBIRD-ETC

BPC: 1.274. Result ID: 698d945f8b517eb4332277aa.

[Reporting paper](<https://www.sota2.com/research/paper/68fa690b1e209131cea1faf6.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa690b1e209131cea1faf6.md>)

[Implementation](<http://goo.gle/bigbird-transformer>)

Recorded evaluation configuration: {"Size":"Large","Sequence Length":"4096"}. This record does not assert independent reproduction.

### Longformer

BPC: 1.358. Result ID: 698d945f8b517eb4332277a6.

[Reporting paper](<https://www.sota2.com/research/paper/68fa690b1e209131cea1faf6.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa68d41e209131cea1e701.md>)

[Implementation](<https://github.com/allenai/longformer>)

Recorded evaluation configuration: {"Size":"Large","Sequence Length":"4096"}. This record does not assert independent reproduction.

### BIGBIRD-ITC

BPC: 1.456. Result ID: 698d945f8b517eb4332277a8.

[Reporting paper](<https://www.sota2.com/research/paper/68fa690b1e209131cea1faf6.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa690b1e209131cea1faf6.md>)

[Implementation](<http://goo.gle/bigbird-transformer>)

Recorded evaluation configuration: {"Size":"Large","Sequence Length":"4096"}. This record does not assert independent reproduction.

### ROBERTa

BPC: 1.496. Result ID: 698d945f8b517eb4332277a4.

[Reporting paper](<https://www.sota2.com/research/paper/68fa690b1e209131cea1faf6.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa66131e209131cea0e18b.md>)

[Implementation](<https://github.com/pytorch/fairseq>)

Recorded evaluation configuration: {"Size":"Large","Sequence Length":"512"}. This record does not assert independent reproduction.

### BIGBIRD-ETC

BPC: 1.611. Result ID: 698d945f8b517eb4332277a9.

[Reporting paper](<https://www.sota2.com/research/paper/68fa690b1e209131cea1faf6.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa690b1e209131cea1faf6.md>)

[Implementation](<http://goo.gle/bigbird-transformer>)

Recorded evaluation configuration: {"Size":"Base","Sequence Length":"4096"}. This record does not assert independent reproduction.

### BIGBIRD-ITC

BPC: 1.678. Result ID: 698d945f8b517eb4332277a7.

[Reporting paper](<https://www.sota2.com/research/paper/68fa690b1e209131cea1faf6.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa690b1e209131cea1faf6.md>)

[Implementation](<http://goo.gle/bigbird-transformer>)

Recorded evaluation configuration: {"Size":"Base","Sequence Length":"4096"}. This record does not assert independent reproduction.

### Longformer

BPC: 1.705. Result ID: 698d945f8b517eb4332277a5.

[Reporting paper](<https://www.sota2.com/research/paper/68fa690b1e209131cea1faf6.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa68d41e209131cea1e701.md>)

[Implementation](<https://github.com/allenai/longformer>)

Recorded evaluation configuration: {"Size":"Base","Sequence Length":"4096"}. This record does not assert independent reproduction.

### ROBERTa

BPC: 1.846. Result ID: 698d945f8b517eb4332277a3.

[Reporting paper](<https://www.sota2.com/research/paper/68fa690b1e209131cea1faf6.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa66131e209131cea0e18b.md>)

[Implementation](<https://github.com/pytorch/fairseq>)

Recorded evaluation configuration: {"Size":"Base","Sequence Length":"512"}. This record does not assert independent reproduction.

Page 1 of 1. This is an index page, not the full collection.