Natural Language Understanding on GLUE 1.0 (test)
97.8SST-2 (Acc)Human Performance
Evaluation results
| Method | Links | |||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Human Performance2019.04 | 97.8 | 66.4 | 86.3 | 80.8 | 92.7 | 92.6 | 59.5 | 80.4 | 92 | 92.8 | 91.2 | 93.6 | 95.9 | — | 87.1 | — | — | — | — | |
| LoRA-α*Backbone=DeBERTa-v3-base-184M, Base learning rate=1 × 10−4, Rank=8, Hyperparameter sweep=0.1×–10× sweep over α2026.06 | 96.44 | 70.89 | — | — | — | — | — | — | — | — | 94.05 | 87 | — | — | 89.09 | 90.75 | 90.44 | 92.06 | 91.11 | |
| LoRA-η*Backbone=DeBERTa-v3-base-184M, Base learning rate=1 × 10−4, Rank=8, Hyperparameter sweep=0.1×–10× sweep over η2026.06 | 96.21 | 69.27 | — | — | — | — | — | — | — | — | 94.05 | 84.47 | — | — | 88.53 | 90.69 | 91.05 | 91.76 | 90.72 | |
| Snorkel MeTaLensemble=true2019.04 | 96.2 | 63.8 | 91.5 | 88.5 | 90.1 | 89.7 | 73.1 | 89.9 | 87.6 | 87.2 | 93.9 | 80.9 | 65.1 | 39.9 | 83.2 | — | — | — | — | |
| LoRABackbone=DeBERTa-v3-base-184M, Base learning rate=1 × 10−4, Rank=82026.06 | 95.87 | 63.56 | — | — | — | — | — | — | — | — | 93.88 | 50.18 | — | — | 81.94 | 90.23 | 84.06 | 90.55 | 87.2 | |
| LoRA-αBackbone=DeBERTa-v3-base-184M, Base learning rate=1 × 10−4, Rank=8, Hyperparameter sweep=None2026.06 | 95.75 | 70.89 | — | — | — | — | — | — | — | — | 94.03 | 80.86 | — | — | 87.97 | 90.43 | 89.95 | 91.74 | 90.12 | |
| RsLoRABackbone=DeBERTa-v3-base-184M, Base learning rate=1 × 10−4, Rank=82026.06 | 95.64 | 64.85 | — | — | — | — | — | — | — | — | 93.97 | 60.31 | — | — | 83.76 | 90.33 | 86.38 | 90.26 | 88.37 | |
| PiSSABackbone=DeBERTa-v3-base-184M, Base learning rate=1 × 10−4, Rank=82026.06 | 95.64 | 65.06 | — | — | — | — | — | — | — | — | 93.84 | 74.36 | — | — | 86.09 | 90.38 | 89.21 | 91.35 | 88.9 | |
| MT-DNN2019.04 | 95.6 | 61.5 | 90 | 86.7 | 88.3 | 87.7 | 72.4 | 89.6 | 86.7 | 86 | — | 75.5 | 65.1 | 40.3 | 82.2 | — | — | — | — | |
| MT-DNNKDbackbone=BERT-Large2019.04 | 95.6 | 65.4 | 91.1 | 88.2 | 89.6 | 89 | 72.7 | 89.6 | 87.5 | 86.7 | 96 | 85.1 | 65.1 | 42.8 | 83.7 | — | — | — | — | |
| AutoBERT-Zero-Large#Params=318M, Infer FLOPs=6.8e102021.07 | 95.4 | 63.8 | 90.7 | — | 90.1 | — | 72.1 | — | 87.7 | 87.1 | 93.6 | 80.4 | — | — | 84.5 | — | — | — | — | |
| LoRA+Backbone=DeBERTa-v3-base-184M, Base learning rate=1 × 10−4, Rank=82026.06 | 95.32 | 64.79 | — | — | — | — | — | — | — | — | 94.32 | 65.37 | — | — | 84.73 | 90.37 | 87.54 | 90.93 | 89.2 | |
| LoRAMBackbone=DeBERTa-v3-base-184M, Base learning rate=1 × 10−4, Rank=82026.06 | 95.29 | 65.53 | — | — | — | — | — | — | — | — | 94.08 | 74.72 | — | — | 86.44 | 90.34 | 89.95 | 91.7 | 89.93 | |
| ALICE2019.04 | 95.2 | 63.5 | 91.8 | 89 | 89.8 | 88.8 | 74 | 90.4 | 87.9 | 87.4 | 95.7 | 80.9 | 65.1 | 40.7 | 83.3 | — | — | — | — | |
| AutoBERT-Zero*#Params=104M, Infer FLOPs=2.3e10, initialization=supernet2021.07 | 95.1 | 59.5 | 90.5 | — | 88.9 | — | 72.8 | — | 86.1 | 86 | 92.1 | 80.2 | — | — | 83.5 | — | — | — | — | |
| BERT LARGE2019.04 | 94.9 | 60.5 | 89.3 | 85.4 | 87.6 | 86.5 | 72.1 | 89.3 | 86.7 | 85.9 | 92.7 | 70.1 | 65.1 | 39.6 | 80.5 | — | — | — | — | |
| BERT-Large#Params=340M, Infer FLOPs=8.7e102021.07 | 94.9 | 60.5 | 89.3 | — | 86.5 | — | 72.1 | — | 86.7 | 85.9 | 92.7 | 70.1 | — | — | 82.1 | — | — | — | — | |
| Roberta-base#Params=110M, Infer FLOPs=2.9e102021.07 | 94.6 | 50.5 | 90 | — | 88.1 | — | 70.9 | — | 86 | 85.4 | 92.5 | 73 | — | — | 81.1 | — | — | — | — | |
| AutoBERT-Zero#Params=104M, Infer FLOPs=2.3e102021.07 | 94.6 | 55.9 | 89.5 | — | 88.3 | — | 71.8 | — | 85.4 | 84.9 | 91.2 | 77.8 | — | — | 82.2 | — | — | — | — | |
| Full Fine-TuningBackbone=DeBERTa-v3-base-184M, Base learning rate=1 × 10−4, Rank=N/A2026.06 | 93.57 | 67.26 | — | — | — | — | — | — | — | — | 92.8 | 83.75 | — | — | 86.69 | 88.31 | 89.46 | 91.52 | 86.87 | |
| BERT-base#Params=110M, Infer FLOPs=2.9e102021.07 | 93.5 | 52.1 | 88.9 | — | 85.8 | — | 71.2 | — | 84.6 | 83.4 | 90.5 | 66.4 | — | — | 79.6 | — | — | — | — | |
| ConvBERT-base#Params=106M, Infer FLOPs=2.7e102021.07 | 93.5 | 53.7 | 89.3 | — | 86.1 | — | 71.3 | — | 84.6 | 83.6 | 90.1 | 72.1 | — | — | 80.5 | — | — | — | — | |
| GPT on STILTS2019.04 | 93.1 | 47.2 | 87.7 | 83.7 | 85.3 | 84.8 | 70.1 | 88.1 | 80.8 | 80.6 | — | 69.1 | 65.1 | 29.4 | 76.9 | — | — | — | — | |
| DynaBERT-base#Params=110M, Infer FLOPs=2.9e102021.07 | 93 | 54.9 | 87.9 | — | 84.4 | — | 72.1 | — | 84.5 | 84.1 | 91.3 | 69.9 | — | — | 80.2 | — | — | — | — | |
| Singletask Pretrain Transformer2019.04 | 91.3 | 45.4 | 82.3 | 75.7 | 82 | 80 | 70.3 | 88.5 | 82.1 | 81.4 | 87.4 | 56 | 53.4 | 29.8 | 72.8 | — | — | — | — | |
| GPT#Params=117M, Infer FLOPs=3.0e102021.07 | 91.3 | 45.4 | 82.3 | — | 82 | — | 70.3 | — | 82.1 | 81.4 | 88.1 | 56 | — | — | 75.4 | — | — | — | — | |
| Fast-R2D2*Para.=52M (Encoder) / 10M (Parser), tree_source=top-down parser2022.03 | 90.71 | 40.11 | — | — | — | — | — | 84.32 | 69.64 | 69.57 | — | — | — | — | — | — | — | — | — | |
| BiLSTM+ELMo+Attn2019.04 | 90.4 | 36 | 84.9 | 77.9 | 75.1 | 73.3 | 64.8 | 84.7 | 76.4 | 76.1 | 79.8 | 56.8 | 65.1 | 26.5 | 70 | — | — | — | — | |
| BERT (12L)Para.=116M2022.03 | 90.25 | 40.72 | — | — | — | — | — | 87.13 | 80 | 80.41 | — | — | — | — | — | — | — | — | — | |
| Fast-R2D2Para.=52M (Encoder) / 10M (Parser), tree_source=R2D2 encoder2022.03 | 90.25 | 38.45 | — | — | — | — | — | 84.35 | 69.36 | 68.8 | — | — | — | — | — | — | — | — | — | |
| R2D2Para.=52M2022.03 | 89.33 | 34.79 | — | — | — | — | — | 84.27 | 69.35 | 68.72 | — | — | — | — | — | — | — | — | — | |
| Fast-R2D2*Para.=52M (Encoder) / 10M (Parser), fine-tuned without Lbilm=true, tree_source=top-down parser2022.03 | 88.3 | 10.14 | — | — | — | — | — | 84.07 | 69.36 | 69.11 | — | — | — | — | — | — | — | — | — | |
| Fast-R2D2Para.=52M (Encoder) / 10M (Parser), fine-tuned without Lbilm=true, inference_type=standard (dagger variant)2022.03 | 87.5 | 8.67 | — | — | — | — | — | 83.97 | 69.53 | 69.5 | — | — | — | — | — | — | — | — | — | |
| BERT (4L)Para.=52M2022.03 | 84.98 | 17.07 | — | — | — | — | — | 84.01 | 73.73 | 74.63 | — | — | — | — | — | — | — | — | — | |
| Imagine-DeBERTa-v3-Lmode=zero-shot2026.03 | 81.1 | — | — | — | — | — | — | — | — | — | — | 53.8 | — | — | — | 33.3 | — | — | — | |
| CAR-DeBERTa-v3-Lmode=zero-shot2026.03 | 56.2 | — | — | — | — | — | — | — | — | — | — | 54.9 | — | — | — | 32 | — | — | — | |
| DeBERTa-v3-Lmode=zero-shot2026.03 | 49.1 | — | — | — | — | — | — | — | — | — | — | 50.5 | — | — | — | 32 | — | — | — |