Linguistic Minimal Pair Scoring on BLiMP
88.9Overall AccuracyHead Ensemble
Evaluation Results
| Method | Links | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Head Ensemblebackbone=ROBERTa-base, scoring=RTD2022.05 | 88.9 | 97 | 83.3 | 86.9 | 81.6 | 93.7 | 88.1 | 88.8 | 95.5 | 83 | 94.2 | 84.2 | 92.7 | — | |
| Human Baselinesource=Warstadt et al. (2020a)2022.05 | 88.6 | 97.5 | 90 | 87.3 | 83.9 | 92.2 | 85 | 86.9 | 97 | 84.9 | 88.1 | 86.6 | 90.9 | — | |
| Head Ensemblebackbone=ROBERTa-base, scoring=HoM2022.05 | 87.8 | 96.3 | 79.6 | 87.6 | 82.6 | 93.6 | 84.9 | 90.4 | 94.3 | 83 | 94.6 | 80.6 | 92.8 | — | |
| RoBERTa (large)Metric=PLL-word-l2r2023.05 | 87.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Phenomenon Headbackbone=ROBERTa-base, scoring=RTD2022.05 | 86.8 | 97.6 | 80.7 | 90.3 | 80.7 | 91.9 | 90.4 | 86.8 | 95.9 | 78.9 | 91.3 | 82.9 | 90.4 | — | |
| RoBERTa (base)Metric=PLL-word-l2r2023.05 | 86.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ROBERTa-largesource=Salazar et al. (2020)2022.05 | 86.5 | 97.8 | 84.6 | 79.1 | 84.1 | 96.8 | 90.8 | 88.9 | 96.8 | 83.4 | 85.5 | 70.2 | 91.4 | — | |
| Phenomenon Headbackbone=ROBERTa-base, scoring=HoM2022.05 | 86.5 | 97.6 | 79.9 | 90.5 | 80.7 | 91.6 | 89.9 | 87.1 | 95.9 | 78.9 | 91.1 | 83.4 | 90.2 | — | |
| RoBERTa (large)Model size=large2022.05 | 86.5 | 97.8 | 84.6 | 79.1 | 84.1 | 96.8 | 90.8 | — | 96.8 | 83.4 | 85.5 | 70.2 | 91.4 | 88.9 | |
| RoBERTa (large)Metric=PLL-original2023.05 | 86.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| RoBERTa (large)Metric=PLL-whole-word2023.05 | 85.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Head Ensemblebackbone=BERT-base, scoring=RTD2022.05 | 85.8 | 93.9 | 82.5 | 85.6 | 77 | 96.3 | 88.1 | 80.7 | 95.7 | 77 | 92.5 | 83.8 | 88.9 | — | |
| ROBERTa-basesource=Salazar et al. (2020)2022.05 | 85.4 | 97.3 | 83.5 | 77.8 | 81.9 | 97 | 91.4 | 90.1 | 96.2 | 80.7 | 81 | 69.8 | 91.9 | — | |
| RoBERTa (base)Model size=base2022.05 | 85.4 | 97.3 | 83.5 | 77.8 | 81.9 | 97 | 91.4 | — | 96.2 | 80.7 | 81 | 69.8 | 91.9 | 90.1 | |
| RoBERTa (base)Metric=PLL-original2023.05 | 85.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| RoBERTa (base)Metric=PLL-whole-word2023.05 | 85.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Transcormer (base)Model size=base, Initialization=BERT, Training steps=20K2022.05 | 85 | 98.1 | 80 | 84.8 | 79.2 | 96.2 | 89.3 | — | 96.1 | 76.6 | 87.8 | 75 | 91.7 | 84.6 | |
| BERT (large)Metric=PLL-word-l2r2023.05 | 85 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BERT-largesource=Salazar et al. (2020)2022.05 | 84.8 | 97.2 | 80.7 | 82 | 82.7 | 97.6 | 86.4 | 84.3 | 92.8 | 77 | 83.4 | 72.8 | 91.9 | — | |
| BERT (large)Model size=large2022.05 | 84.8 | 97.2 | 80.7 | 82 | 82.7 | 97.6 | 86.4 | — | 92.8 | 77 | 83.4 | 72.8 | 91.9 | 84.3 | |
| BERT (large)Metric=PLL-original2023.05 | 84.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BERT (base)Metric=PLL-word-l2r2023.05 | 84.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Transcormer (base)Model size=base2022.05 | 84.6 | 98.1 | 80.7 | 83.2 | 80.2 | 96 | 90.7 | — | 95.5 | 74.3 | 85.7 | 73.4 | 91.3 | 84.1 | |
| Head Ensemblebackbone=BERT-base, scoring=HoM2022.05 | 84.3 | 93.3 | 79.9 | 83.5 | 78.6 | 96.4 | 78.4 | 79.5 | 93.8 | 74.4 | 92.5 | 81.7 | 86.8 | — | |
| BERT-basesource=Salazar et al. (2020)2022.05 | 84.2 | 97 | 80 | 82.3 | 79.6 | 97.6 | 89.4 | 83.1 | 96.5 | 73.6 | 84.7 | 71.2 | 92.4 | — | |
| BERT (base)Model size=base2022.05 | 84.2 | 97 | 80 | 82.3 | 79.6 | 97.6 | 89.4 | — | 96.5 | 73.6 | 84.7 | 71.2 | 92.4 | 83.1 | |
| BERT (base)Metric=PLL-original2023.05 | 84.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BERT (base)Metric=PLL-whole-word2023.05 | 83.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GPT2-mediumsource=Salazar et al. (2020)2022.05 | 82.6 | 99.4 | 83.4 | 77.8 | 83 | 96.3 | 86.3 | 81.3 | 94.9 | 71.7 | 74.7 | 74.1 | 88.3 | — | |
| GPT-2Model size=345M2022.05 | 82.6 | 99.4 | 83.4 | 77.8 | 83 | 96.3 | 86.3 | — | 94.9 | 71.7 | 74.7 | 74.1 | 88.3 | 81.3 | |
| BERT (large)Metric=PLL-whole-word2023.05 | 82.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Top Head [11; 10]backbone=ROBERTa-base, scoring=HoM2022.05 | 81.9 | 90.1 | 66 | 84.7 | 71.7 | 91 | 86.7 | 87.1 | 89.5 | 76.8 | 90.7 | 78.4 | 85 | — | |
| Phenomenon Headbackbone=BERT-base, scoring=RTD2022.05 | 81.8 | 94.5 | 75.8 | 80.4 | 79.2 | 96.7 | 89.1 | 75 | 93 | 72.2 | 84.4 | 81.2 | 82 | — | |
| Phenomenon Headbackbone=BERT-base, scoring=HoM2022.05 | 81.7 | 94.9 | 75.9 | 80.4 | 79.2 | 96.7 | 89.1 | 75.9 | 93 | 70.5 | 84.6 | 81.2 | 82.1 | — | |
| GPT2-largesource=Warstadt et al. (2020a)2022.05 | 81.5 | 99.6 | 78.3 | 80.1 | 80.5 | 93.3 | 86.6 | 81.3 | 84.1 | 70.6 | 78.9 | 71.3 | 89 | — | |
| Top Head [11; 10]backbone=ROBERTa-base, scoring=RTD2022.05 | 80.8 | 88.8 | 63.3 | 83.7 | 69.8 | 90.2 | 88.8 | 86.8 | 89.1 | 76.5 | 89 | 78.4 | 83.1 | — | |
| UNIONNumber of parameters=1.3B, Procedural pretraining status=Yes, Pre-training tokens (C4)=6.6B2026.01 | 75.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Top Head [8; 0]backbone=BERT-base, scoring=HoM2022.05 | 75.4 | 86.8 | 75.9 | 63.4 | 79.2 | 83.7 | 72.2 | 67.3 | 90.3 | 70 | 83.1 | 63.5 | 82.1 | — | |
| Top Head [8; 0]backbone=BERT-base, scoring=RTD2022.05 | 75.4 | 86.8 | 75.8 | 63.3 | 79.2 | 83.6 | 72.1 | 67.3 | 90.2 | 70.2 | 83.1 | 63.6 | 82 | — | |
| All Headsbackbone=ROBERTa-base, scoring=HoM2022.05 | 74.3 | 80.6 | 71.2 | 78.6 | 67.8 | 90.6 | 89.9 | 75.5 | 65.1 | 73.4 | 65.1 | 57 | 75.6 | — | |
| All Headsbackbone=ROBERTa-base, scoring=RTD2022.05 | 74.3 | 81.1 | 66.6 | 76.8 | 65.2 | 89.9 | 91.1 | 74.6 | 63.3 | 74.1 | 63.2 | 56.9 | 74.9 | — | |
| No Procedural PretrainingNumber of parameters=1.3B, Procedural pretraining status=No, Pre-training tokens (C4)=6.6B2026.01 | 73.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| UNIONNumber of parameters=350M, Procedural pretraining status=Yes, Pre-training tokens (C4)=1.6B2026.01 | 72.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| No Procedural PretrainingNumber of parameters=350M, Procedural pretraining status=No, Pre-training tokens (C4)=1.6B2026.01 | 71.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LSTMsource=Warstadt et al. (2020a)2022.05 | 69.8 | 91.7 | 73.2 | 73.5 | 67 | 85.4 | 67.6 | 73.9 | 89.1 | 46.6 | 51.7 | 64.5 | 80.1 | — | |
| Transformer-XLsource=Warstadt et al. (2020a)2022.05 | 69.6 | 94.1 | 72.2 | 74.7 | 71.5 | 83 | 77.2 | 66.6 | 78.2 | 48.4 | 55.2 | 69.3 | 76 | — | |
| All Headsbackbone=BERT-base, scoring=RTD2022.05 | 65.3 | 77.8 | 68.5 | 63.2 | 63.6 | 86 | 73.4 | 48.4 | 91 | 51.3 | 62 | 51.2 | 72.6 | — | |
| All Headsbackbone=BERT-base, scoring=HoM2022.05 | 64.8 | 79.6 | 69.1 | 63.9 | 62.6 | 86.2 | 70.7 | 47.3 | 90.7 | 49.5 | 61.1 | 50 | 72 | — | |
| 5-gramsource=Warstadt et al. (2020a)2022.05 | 61.2 | 47.9 | 71.9 | 64.4 | 68.5 | 70 | 36.9 | 60.2 | 79.5 | 57.2 | 45.5 | 53.5 | 60.3 | — |