Machine Reading Comprehension on RACE (test)
95.4RACE Accuracy (Medium)Human Performance (Ceiling)
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| Human Performance (Ceiling)2019.01 | 95.4 | 94.2 | 94.5 | — | — | |
| CeilingDescription=Human Performance, Model Scope=Single model2019.08 | 95.4 | 94.2 | 94.5 | — | — | |
| Megatron-3.9B ensembleensemble=true, fine-tuning=true2019.09 | 93.1 | 90 | — | — | 90.9 | |
| Megatron-BERTInference Mode=ensemble, Source=leaderboard2020.01 | 93.1 | 90 | — | 90.9 | — | |
| ALBERTxxlargeplus=SC + TL2022.03 | 92.8 | 89.8 | — | 90.7 | — | |
| ALBERT_xxlarge+DUMAInference Mode=ensemble, Source=our model2020.01 | 92.6 | 88.7 | — | 89.8 | — | |
| Megatron-3.9Btrained tokens ratio=1, fine-tuning=true2019.09 | 91.8 | 88.6 | — | — | 89.5 | |
| Megatron-BERTInference Mode=single, Source=leaderboard2020.01 | 91.8 | 88.6 | — | 89.5 | — | |
| Megatron-BERT-3.9B2022.03 | 91.8 | 88.6 | — | 89.5 | — | |
| POI-Netbackbone=ALBERTxxlarge2022.03 | 91.5 | 86.8 | — | 88.3 | — | |
| POI-NetBackbone=ALBERT-xxlarge2022.03 | 91.5 | 86.8 | — | 88.3 | — | |
| ALBERTModel type=Ensemble2019.09 | 91.2 | 88.6 | — | 89.4 | — | |
| ALBERT ensembleensemble=true, fine-tuning=true2019.09 | 91.2 | 88.6 | — | — | 89.4 | |
| ALBERTInference Mode=ensemble, Source=leaderboard2020.01 | 91.2 | 88.6 | — | 89.4 | — | |
| ALBERT_xxlarge+DUMAInference Mode=single, Source=our model2020.01 | 90.9 | 86.7 | — | 88 | — | |
| ALBERTxxlargeplus=DUMA2022.03 | 90.9 | 86.7 | — | 88 | — | |
| ALBERTxxlarge + DUMABackbone=ALBERT-xxlarge2022.03 | 90.9 | 86.7 | — | 88 | — | |
| Megatron-1.3Btrained tokens ratio=1, fine-tuning=true2019.09 | 90.4 | 86.1 | — | — | 87.3 | |
| ALBERTxxlargererun=true2022.03 | 89.2 | 85.4 | — | 86.5 | — | |
| ALBERTxxlargeBackbone=ALBERT-xxlarge, rerun=true2022.03 | 89.2 | 85.4 | — | 86.5 | — | |
| RoBERTa + MMMSource=leaderboard2020.01 | 89.1 | 83.3 | — | 85 | — | |
| RoBERTalargeplus=MMM2022.03 | 89.1 | 83.3 | — | 85 | — | |
| RoBERTalarge+MMMBackbone=RoBERTa-large2022.03 | 89.1 | 83.3 | — | 85 | — | |
| ALBERTModel type=Single model, Training steps=1.5M2019.09 | 89 | 85.5 | — | 86.5 | — | |
| ALBERTtrained tokens ratio=3, fine-tuning=true2019.09 | 89 | 85.5 | — | — | 86.5 | |
| ALBERTInference Mode=single, Source=leaderboard2020.01 | 89 | 85.5 | — | 86.5 | — | |
| ALBERT_xxlargeSource=our model, Reproduction=re-run baseline2020.01 | 89 | 85.5 | — | 86.6 | — | |
| ALBERTxxlargeBackbone=ALBERT-xxlarge2022.03 | 89 | 85.5 | — | 86.5 | — | |
| B10-10-10Scale=Large, Layout=B10-10-102020.06 | 88.8 | 84.4 | 85.7 | — | — | |
| XLNetArchitecture=24-layer2019.06 | 88.6 | 84 | — | 85.4 | — | |
| XLNettrained tokens ratio=2, fine-tuning=true2019.09 | 88.6 | 84 | — | — | 85.4 | |
| XLNet_LargeScale=Large2020.06 | 88.6 | 84 | 85.4 | — | — | |
| DCMN+Model type=Ensemble2019.09 | 88.5 | 82.3 | — | 84.1 | — | |
| B8-8-8Scale=Large, Layout=B8-8-82020.06 | 88.4 | 83.9 | 85.2 | — | — | |
| ALBERTModel type=Single model, Training steps=1M2019.09 | 88.2 | 85.1 | — | 86 | — | |
| FLOATERModel size=large model, Evaluation setting=Single model on test2020.03 | 87.1 | 81.7 | — | 83.3 | — | |
| Megatron-336Mtrained tokens ratio=1, fine-tuning=true2019.09 | 86.9 | 81.5 | — | — | 83 | |
| RoBERTamodel_type=Single model2019.07 | 86.5 | 81.3 | 83.2 | — | — | |
| RoBERTaArchitecture=24-layer2019.06 | 86.5 | 81.8 | — | 83.2 | — | |
| RoBERTaModel type=Single model2019.09 | 86.5 | 81.3 | — | 83.2 | — | |
| RoBERTatrained tokens ratio=2, fine-tuning=true2019.09 | 86.5 | 81.8 | — | — | 83.2 | |
| RoBERTaModel size=large model, Evaluation setting=Single model on test2020.03 | 86.5 | 81.3 | — | 82.8 | — | |
| ROBERTA_LargeScale=Large2020.06 | 86.5 | 81.3 | 83.2 | — | — | |
| XLNet_xxlarge + DCMN+Source=publication2020.01 | 86.5 | 81.3 | — | 82.8 | — | |
| XLNetlargeplus=DCMN+2022.03 | 86.5 | 81.3 | — | 82.8 | — | |
| RoBERTalarge2022.03 | 86.5 | 81.8 | — | 83.2 | — | |
| RoBERTalargeBackbone=RoBERTa-large2022.03 | 86.5 | 81.8 | — | 83.2 | — | |
| XLNetModel type=Single model2019.09 | 85.5 | 80.2 | — | 81.8 | — | |
| XLNetSource=publication2020.01 | 85.5 | 80.2 | — | 81.8 | — | |
| XLNetlarge2022.03 | 85.5 | 80.2 | — | 81.8 | — | |
| XLNet_LARGEmodel_type=Single model2019.07 | 85.4 | 80.2 | 81.7 | — | — | |
| Human performancenumber of models=12018.10 | 85.1 | 69.4 | 73.3 | — | — | |
| Human Performance (Turkers)2019.01 | 85.1 | 69.4 | 73.3 | — | — | |
| TurkersDescription=Human Performance, Model Scope=Single model2019.08 | 85.1 | 69.4 | 73.3 | — | — | |
| B6-6-6Scale=Base, Layout=B6-6-62020.06 | 83.4 | 78.2 | 79.7 | — | — | |
| B6-3x2-3x2Scale=Base, Layout=B6-3x2-3x22020.06 | 82 | 77.5 | 78.8 | — | — | |
| B4-4-4Scale=Base, Layout=B4-4-42020.06 | 80 | 74.6 | 76.2 | — | — | |
| MPNetModel setting=BERT_BASE, Pre-training data size=160GB2020.04 | 79.7 | 74.5 | 76.1 | — | — | |
| DMN+_largeEncoder=BERTlarge, Ensemble=false2019.01 | 79.5 | 71.8 | 74.1 | — | — | |
| BERT+DCMNEnsemble=true, Architecture=24-layer2019.06 | 79.5 | 71.8 | — | 74.1 | — | |
| SG-NetCategory=Leaderboard, Model Scope=Single model2019.08 | 78.8 | 72.2 | 74.2 | — | — | |
| SG-NetBackbone=BERT-large2022.03 | 78.8 | 72.2 | — | 74.2 | — | |
| OCNEnsemble=true2019.01 | 78.4 | 71.5 | 73.5 | — | — | |
| BaselineCategory=Leaderboard, Model Scope=Single model2019.08 | 78.4 | 70.4 | 72.6 | — | — | |
| DMN_largeEncoder=BERTlarge, Ensemble=false2019.01 | 77.6 | 70.1 | 72.3 | — | — | |
| DCMNCategory=Leaderboard, Model Scope=Single model2019.08 | 77.6 | 70.1 | 72.3 | — | — | |
| MPNetModel setting=BERT_BASE, Pre-training data size=16GB (Wikipedia and BooksCorpus)2020.04 | 76.8 | 67.7 | 70.4 | — | — | |
| OCNCategory=Leaderboard, Model Scope=Single model2019.08 | 76.7 | 69.6 | 71.7 | — | — | |
| OCNSource=publication2020.01 | 76.7 | 69.6 | — | 71.7 | — | |
| BERT_LARGEmodel_type=Single model2019.07 | 76.6 | 70.1 | 72 | — | — | |
| BERT_largeEncoder=BERTlarge2019.01 | 76.6 | 70.1 | 72 | — | — | |
| BERT_LARGECategory=Leaderboard, Model Scope=Single model2019.08 | 76.6 | 70.1 | 72 | — | — | |
| BERTArchitecture=24-layer2019.06 | 76.6 | 70.1 | — | 72 | — | |
| BERT-largeModel type=Single model2019.09 | 76.6 | 70.1 | — | 72 | — | |
| BERTlarge2022.03 | 76.6 | 70.1 | — | 72 | — | |
| BERTlargeBackbone=BERT-large2022.03 | 76.6 | 70.1 | — | 72 | — | |
| POI-Netbackbone=ALBERTbase2022.03 | 75.7 | 69 | — | 71 | — | |
| POI-NetBackbone=ALBERT-base2022.03 | 75.7 | 69 | — | 71 | — | |
| DMN_baseEncoder=BERTbase, Ensemble=false2019.01 | 72.3 | 64.2 | 66.5 | — | — | |
| ALBERTbasererun=true2022.03 | 72.1 | 65.2 | — | 67.2 | — | |
| ALBERTbaseBackbone=ALBERT-base, rerun=true2022.03 | 72.1 | 65.2 | — | 67.2 | — | |
| GPT + Self-Assessment + Highlighting + Back and Forth Readingnumber of models=9, Self-Assessment=true, Highlighting=true, Back and Forth Reading=true2018.10 | 72 | 64.5 | 66.7 | — | — | |
| BERTModel setting=BERT_BASE, Pre-training data size=16GB (Wikipedia and BooksCorpus)2020.04 | 71.7 | 62.3 | 65 | — | — | |
| BERT_baseEncoder=BERTbase2019.01 | 71.1 | 62.3 | 65 | — | — | |
| BERTbase2022.03 | 71.1 | 62.3 | — | 65 | — | |
| BERTbaseBackbone=BERT-base2022.03 | 71.1 | 62.3 | — | 65 | — | |
| GPT + Self-Assessment + Highlighting + Back and Forth Readingnumber of models=2, Self-Assessment=true, Highlighting=true, Back and Forth Reading=true2018.10 | 70.9 | 63.2 | 65.4 | — | — | |
| MPNet_BaseScale=Base2020.06 | 70.3 | 76.3 | 72 | — | — | |
| GPT + Self-Assessment + Highlightingnumber of models=1, Self-Assessment=true, Highlighting=true2018.10 | 69.2 | 61.5 | 63.8 | — | — | |
| RSM2019.01 | 69.2 | 61.5 | 63.8 | — | — | |
| RSMSource=publication2020.01 | 69.2 | 61.5 | — | 63.8 | — | |
| GPT + Highlightingnumber of models=1, Highlighting=true2018.10 | 67.4 | 61.5 | 63.2 | — | — | |
| GPT + Back and Forth Readingnumber of models=2, Back and Forth Reading=true2018.10 | 67.3 | 60.7 | 62.6 | — | — | |
| MMNnumber of models=92018.10 | 64.7 | 55.5 | 58.2 | — | — | |
| GPT (our implementation)number of models=92018.10 | 63.5 | 59.3 | 60.6 | — | — | |
| GPT + Self-Assessmentnumber of models=1, Self-Assessment=true2018.10 | 63.2 | 59.2 | 60.4 | — | — | |
| GPTnumber of models=12018.10 | 62.9 | 57.4 | 59 | — | — | |
| GPT2019.01 | 62.9 | 57.4 | 59 | — | — | |
| GPT2019.06 | 62.9 | 57.4 | — | 59 | — | |
| GPTSource=publication2020.01 | 62.9 | 57.4 | — | 59 | — |