Natural Language Understanding on GLUE (test)
97.9SST-2 AccuracyZ-Code++
Evaluation results
| Method | Links | |||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Z-Code++Architecture=Encoder-Decoder, Fine-tuning Protocol=jointly fine-tuned2022.08 | 97.9 | — | 69.2 | 95.1 | 90.7 | 91.2 | — | — | — | — | — | 91 | 90.9 | — | — | — | 89.6 | 90 | — | — | 89.33 | — | — | — | — | — | — | — | — | — | — | |
| Human Performance2019.01 | 97.8 | 86.3 | 66.4 | 91.2 | 93.6 | 92.7 | 87.1 | 80.8 | 92.6 | 80.4 | 59.5 | 92 | 92.8 | 95.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Human Baseline2019.08 | 97.8 | 80.8 | 66.4 | 91.2 | 93.6 | 92.7 | — | 86.3 | 92.6 | 80.4 | 59.5 | 92 | 92.8 | 95.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Human Baselines2021.01 | 97.8 | 86.3 | 66.4 | 91.2 | 93.6 | 92.7 | — | 80.8 | 92.6 | 80.4 | 59.5 | 92 | 92.8 | — | — | — | — | — | — | — | 87.1 | — | — | — | — | — | — | — | — | — | — | |
| MT-DNN-SMART+Model Type=Ensemble, Configuration=Mixed results from ensemble and single of MT-DNN SMART and with data augmentation, #param=356M2019.11 | 97.5 | 91.6 | 69.5 | 99.2 | 89.7 | 92.9 | 89.9 | 93.7 | 92.5 | 73.9 | 90.2 | 91 | 90.8 | 94.5 | 50.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SMART_ROBERTAModel Type=Single Model, #param=356M2019.11 | 97.5 | 91.6 | 65.1 | 95.4 | 87.9 | 92.9 | 88.4 | 93.7 | 92.5 | 74 | 90.1 | 91 | 90.8 | 91.8 | 50.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| B10-10-10H1024Evaluation Protocol=Multi-task & Ensemble, Layout=B10-10-10, Hidden Size=10242020.06 | 97.5 | — | 70.5 | 95.8 | 90 | — | 89.7 | — | — | — | — | — | — | 94.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DeBERTaBackbone=DeBERTa, Number of trainable parameters=3 * 10^92021.01 | 97.5 | 94 | 71.5 | 90.2 | 93.2 | 92.9 | — | 92 | 92.6 | 90.8 | 76.2 | 91.9 | 91.6 | — | — | — | — | — | — | — | 90.8 | — | — | — | — | — | — | — | — | — | — | |
| ROBERTa with EnsembleBackbone=ROBERTa-LARGE, Ensemble=5 to 7 models per task, Intermediate task fine-tuning=true2020.09 | 97.5 | 91.6 | 67.8 | 95.4 | 87.9 | — | 87.3 | — | — | — | 74 | 91 | 90.8 | — | — | 92.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| T5-11BParameters=11 billion2019.10 | 97.5 | 90.4 | 71.6 | 96.9 | 92.8 | 93.1 | — | 92.8 | 92.8 | 75.1 | 90.6 | 92.2 | 91.9 | 94.5 | — | — | — | — | — | — | 90.3 | — | — | — | — | — | — | — | — | — | — | |
| T5-3BParameters=3 billion2019.10 | 97.4 | 90 | 67.1 | 96.3 | 91.1 | 90.6 | — | 92.5 | 89.8 | 74.4 | 89.7 | 91.4 | 91.2 | 89.7 | — | — | — | — | — | — | 88.5 | — | — | — | — | — | — | — | — | — | — | |
| XELECTRA-lModel Size=large2022.04 | 97.36 | 92.33 | 70.51 | 94.97 | 88.45 | — | — | — | — | — | — | — | — | — | — | 92.04 | — | 91.4 | — | — | — | 91.03 | — | — | — | — | — | — | — | — | — | |
| B10-10-10H1024Evaluation Protocol=Single Task & Single Model, Layout=B10-10-10, Hidden Size=10242020.06 | 97.2 | — | 68.9 | 95.5 | 86.5 | — | 85.4 | — | — | — | — | — | — | 65.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ALICEModel Type=Ensemble, #param=340M2019.11 | 97.1 | 91.5 | 69.2 | 99.2 | 87.3 | 92.7 | 89 | 93.6 | 92.3 | 74.4 | 90.7 | 90.7 | 90.2 | 89.7 | 47.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ALBERTModel Type=Ensemble, #param=235M*2019.11 | 97.1 | 91.2 | 69.1 | 99.2 | 89.2 | 92.5 | 89.4 | 93.4 | 92 | 74.2 | 90.5 | 91.3 | 91 | 91.8 | 50.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| T5Model Type=Single Model, #param=11,000M2019.11 | 97.1 | 89.2 | 70.8 | 96.7 | 92.5 | 92.5 | 89.7 | 91.9 | 92.1 | 74.6 | 90.4 | 92 | 91.7 | 93.2 | 53.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| StructBERT_RoBERTa ensembleEnsemble=true, Model fusion=StructBERT + RoBERTa2019.08 | 97.1 | 91.5 | 69.2 | 99.2 | 87.3 | 92.8 | 89 | 93.6 | 92.4 | 90.7 | 74.4 | 90.7 | 90.3 | 89.7 | 47.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| XLNet*Learning Strategy=Multi-task, Model Configuration=Ensemble2019.06 | 97.1 | — | 70.2 | 99 | 88.5 | — | — | — | — | — | — | 90.9 | 90.9 | 92.5 | — | 93 | 92.9 | 90.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ALBERTevaluation_protocol=ensemble2019.09 | 97.1 | 93.4 | 69.1 | 99.2 | 89.2 | 92.5 | 89.4 | — | — | — | 90.5 | 91.3 | — | 91.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ALBERTTrain FLOPs=3.1e22 (10x)2020.03 | 97.1 | 91.2 | 69.1 | — | 89.2 | — | 89 | — | — | — | 90.5 | — | — | 91.8 | — | 92 | — | — | — | 91.3 | — | — | — | — | — | — | — | — | — | — | — | |
| XLNetTrain FLOPs=3.9e21 (1.26x)2020.03 | 97.1 | 90.5 | 70.2 | — | 88.5 | — | 89.1 | — | — | — | 90.4 | — | — | 92.5 | — | 92.6 | — | — | — | 90.9 | — | — | — | — | — | — | — | — | — | — | — | |
| ELECTRATrain FLOPs=3.1e21 (1x)2020.03 | 97.1 | 90.7 | 71.7 | 95.8 | 89.8 | — | 89.5 | — | — | — | 90.8 | — | — | 92.5 | — | 92.5 | — | — | — | 91.3 | 89.4 | — | — | — | — | — | — | — | — | — | — | |
| ELECTRA-LargeEvaluation Protocol=Multi-task & Ensemble2020.06 | 97.1 | — | 71.7 | 95.8 | 89.8 | — | 89.4 | — | — | — | — | — | — | 91.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Previous best2019.10 | 97.1 | 91.5 | 69.2 | 99.2 | 89.2 | 92.7 | — | 93.6 | 92.3 | 74.8 | 90.7 | 91.3 | 91 | 91.8 | — | — | — | — | — | — | 89.4 | — | — | — | — | — | — | — | — | — | — | |
| B8-8-8H1024Evaluation Protocol=Single Task & Single Model, Layout=B8-8-8, Hidden Size=10242020.06 | 96.9 | — | 68.3 | 95.1 | 85.3 | — | 85 | — | — | — | — | — | — | 65.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FreeLBModel Type=Ensemble, #param=356M2019.11 | 96.8 | 90.8 | 68 | 98.8 | 88.7 | 92.4 | 88.8 | 93.1 | 92.2 | 74.8 | 90.3 | 91.1 | 90.7 | 89 | 50.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| XLNetMode=Ensembles, Architecture=24-layer2019.07 | 96.8 | — | 67.8 | 98.6 | 86.3 | — | 88.4 | — | — | — | 90.3 | 90.2 | 89.8 | 90.4 | — | 91.6 | 93 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| XLNet ensembleEnsemble=true2019.08 | 96.8 | 90.7 | 67.8 | 98.6 | 86.3 | 91.6 | 88.4 | 93 | 91.1 | 90.3 | 74.2 | 90.2 | 89.8 | 90.4 | 47.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Adv-RoBERTa ensembleEnsemble=true2019.08 | 96.8 | 90.8 | 68 | 98.8 | 88.7 | 92.4 | 88.8 | 93.1 | 92.2 | 90.3 | 74.8 | 91.1 | 90.7 | 89 | 50.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| XLNetevaluation_protocol=ensemble2019.09 | 96.8 | 93 | 67.8 | 98.6 | 86.3 | 91.6 | 88.4 | — | — | — | 90.3 | 90.2 | — | 90.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Adv-RoBERTaevaluation_protocol=ensemble2019.09 | 96.8 | 93.1 | 68 | 98.8 | 88.7 | 92.4 | 88.8 | — | — | — | 90.3 | 91.1 | — | 89 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MEFT1Backbone=RoBERTalarge, Batch size=32, Sequence length=128, Gradient=vanilla, Precision=FP16, #Param (%)=0.23, Peak Memory (GB)=6.19, Activation Memory (GB)=4.812023.06 | 96.8 | 91.5 | 69.9 | 94.9 | 89.5 | — | 89.6 | — | — | — | — | — | — | — | — | 92.3 | — | 91.5 | — | — | — | 90.3 | — | — | — | — | — | — | — | — | — | |
| MEFT2Backbone=RoBERTalarge, Batch size=32, Sequence length=128, Gradient=vanilla, Precision=FP16, #Param (%)=0.23, Peak Memory (GB)=6.2, Activation Memory (GB)=4.822023.06 | 96.8 | 91.3 | 68.8 | 94.8 | 88.6 | — | 89.3 | — | — | — | — | — | — | — | — | 92.2 | — | 91.4 | — | — | — | 90.6 | — | — | — | — | — | — | — | — | — | |
| RoBERTaModel Type=Ensemble, #param=356M2019.11 | 96.7 | 89.8 | 67.8 | 98.9 | 88.2 | 92.2 | 88.5 | 92.3 | 91.9 | 74.3 | 90.2 | 90.8 | 90.2 | 89 | 48.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| RoBERTaMode=Ensembles, Architecture=24-layer2019.07 | 96.7 | — | 67.8 | 98.9 | 88.2 | — | 88.5 | — | — | — | 90.2 | 90.8 | 90.2 | 89 | — | 92.2 | 92.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| RoBERTa ensembleEnsemble=true2019.08 | 96.7 | 89.8 | 67.8 | 98.9 | 88.2 | 92.2 | 88.5 | 92.3 | 91.9 | 90.2 | 74.3 | 90.8 | 90.2 | 89 | 48.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ROBERTa*Learning Strategy=Multi-task, Model Configuration=Ensemble2019.06 | 96.7 | — | 67.8 | 98.9 | 88.2 | — | — | — | — | — | — | 90.8 | 90.2 | 89 | — | 92.2 | 92.3 | 90.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| RoBERTaevaluation_protocol=ensemble2019.09 | 96.7 | 92.3 | 67.8 | 98.9 | 88.2 | 92.2 | 88.5 | — | — | — | 90.2 | 90.8 | — | 89 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ELECTRA-1.75MTrain FLOPs=3.1e21, Params=330M, Finetuning=single-task2020.03 | 96.7 | 89.2 | — | 95.5 | 86.1 | — | 88.6 | — | — | — | 90.4 | — | — | — | — | 91.7 | — | — | 68.1 | 90.7 | — | — | — | — | — | — | — | — | — | — | — | |
| RoBERTaTrain FLOPs=3.2e21 (1.02x)2020.03 | 96.7 | 89.8 | 67.8 | 95.4 | 88.2 | — | 88.1 | — | — | — | 90.2 | — | — | 89 | — | 91.9 | — | — | — | 90.8 | 88.1 | — | — | — | — | — | — | — | — | — | — | |
| ELECTRA-LargeEvaluation Protocol=Single Task & Single Model2020.06 | 96.7 | — | 68.1 | 95.5 | 86.1 | — | 85.2 | — | — | — | — | — | — | 65.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| RoBERTa-LargeEvaluation Protocol=Multi-task & Ensemble2020.06 | 96.7 | — | 67.8 | 95.4 | 88.2 | — | 88.1 | — | — | — | — | — | — | 89 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| RoBERTaBackbone=RoBERTa, Number of trainable parameters=355 * 10^62021.01 | 96.7 | 92.3 | 67.8 | 95.4 | 88.2 | 92.2 | — | 89.8 | 91.9 | 90.2 | 74.3 | 90.8 | 90.2 | — | — | — | — | — | — | — | 88.1 | — | — | — | — | — | — | — | — | — | — | |
| MEFT1Backbone=RoBERTalarge, Batch size=32, Sequence length=128, Gradient=reversible, Precision=FP32, #Param (%)=0.23, Peak Memory (GB)=3.63, Activation Memory (GB)=2.252023.06 | 96.7 | 91.2 | 66.1 | 94.8 | 90 | — | 88.9 | — | — | — | — | — | — | — | — | 92.4 | — | 90.2 | — | — | — | 90.1 | — | — | — | — | — | — | — | — | — | |
| AdapterPBackbone=RoBERTalarge, Batch size=32, Sequence length=128, Precision=FP16, #Param (%)=0.23, Peak Memory (GB)=6.16, Activation Memory (GB)=4.772023.06 | 96.6 | 89.7 | 67.8 | 94.8 | 80.1 | — | 87.9 | — | — | — | — | — | — | — | — | 91.9 | — | 91.7 | — | — | — | 90.5 | — | — | — | — | — | — | — | — | — | |
| AdapterBackbone=RoBERTa (355M), # para=0.8M2024.10 | 96.6 | 89.7 | 67.8 | 94.8 | 80.1 | — | — | — | — | — | 91.7 | — | — | — | — | 91.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| RoCoFT1-RowBackbone=RoBERTa-Large, TP=0.22M2026.02 | 96.6 | 90 | — | 94.2 | 85.3 | 91.8 | 88.1 | — | — | — | 90.2 | — | — | — | — | — | — | — | 65.7 | 90.7 | — | — | — | — | — | — | — | — | — | — | — | |
| MT-DNNMode=Ensembles, Architecture=24-layer2019.07 | 96.5 | — | 68.4 | 96 | 86.3 | — | 87.6 | — | — | — | 89.9 | 87.9 | 87.4 | 89 | — | 91.1 | 92.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MT-DNN ensembleEnsemble=true2019.08 | 96.5 | 89.5 | 65.4 | 96 | 85.7 | 89.6 | 84.2 | 92.2 | 89 | 89.9 | 73.7 | 87.9 | 87.4 | 65.1 | 42.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MT-DNN*Learning Strategy=Multi-task, Model Configuration=Ensemble2019.06 | 96.5 | — | 68.4 | 96 | 86.3 | — | — | — | — | — | — | 87.9 | 87.4 | 89 | — | 91.1 | 92.7 | 89.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MT-DNNevaluation_protocol=ensemble2019.09 | 96.5 | 92.7 | 68.4 | 96 | 86.3 | 91.1 | 87.6 | — | — | — | 89.9 | 87.9 | — | 89 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| B6-6-6H768Evaluation Protocol=Single Task & Single Model, Layout=B6-6-6, Hidden Size=7682020.06 | 96.5 | — | 68.3 | 94 | 80.4 | — | 84 | — | — | — | — | — | — | 65.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| OK-TransformerLM=RoBERTa2023.05 | 96.44 | 91.91 | 66.89 | 94.71 | 86.28 | 92.19 | 88.49 | 94.24 | 92.36 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 1 | — | — | — | — | — | — | |
| BOFTBackbone=DeBERTaV3-base, d=8, m=2, #Params=0.75M2024.05 | 96.44 | 92.4 | 72.95 | 94.23 | 88.81 | — | 89.89 | — | — | — | 92.1 | — | — | — | — | 91.92 | — | — | — | 90.25 | — | — | — | — | — | — | — | — | — | — | — | |
| Full FTBackbone=RoBERTalarge, Batch size=32, Sequence length=128, Precision=FP16, #Param (%)=100, Peak Memory (GB)=11.47, Activation Memory (GB)=6.052023.06 | 96.4 | 90.9 | 68 | 94.7 | 86.6 | — | 88.9 | — | — | — | — | — | — | — | — | 92.4 | — | 92.2 | — | — | — | 90.2 | — | — | — | — | — | — | — | — | — | |
| FTBackbone=RoBERTa (355M), # para=355M2024.10 | 96.4 | 90.9 | 68 | 94.7 | 86.6 | — | — | — | — | — | 92.2 | — | — | — | — | 92.4 | — | — | — | 90.2 | 88.9 | — | — | — | — | — | — | — | — | — | — | |
| Full TuningBackbone=ROBERTa-large, # Params=355.0M2025.06 | 96.4 | 90.9 | 68 | 94.7 | 86.6 | — | 88.9 | — | — | — | 92.2 | — | — | — | — | 92.4 | — | — | — | 90.2 | — | — | — | — | — | — | — | — | — | — | — | |
| Full FinetuneBackbone=RoBERTa-Large, TP=355M2026.02 | 96.4 | 90.9 | — | 94.7 | 86.6 | 92.3 | 88.9 | — | — | — | 92.2 | — | — | — | — | — | — | — | 68 | 90.2 | — | — | — | — | — | — | — | — | — | — | — | |
| LoRA(r = 4)Backbone=RoBERTa-Large, TP=1.8M2026.02 | 96.4 | 90.7 | — | 94.8 | 89.5 | 92.3 | 89.3 | — | — | — | 91.7 | — | — | — | — | — | — | — | 71 | 90.4 | — | — | — | — | — | — | — | — | — | — | — | |
| WARP (RoBERTa)Backbone=RoBERTa, Number of trainable parameters=< 25,000, Initialization=RTE, STS-B, and MRPC tasks initialized from MNLI parameters2021.01 | 96.3 | 88.2 | 53.9 | 93.5 | 84.3 | 89.5 | — | 83.9 | 88.8 | 87.7 | 68.6 | 88 | 88.2 | — | — | — | — | — | — | — | 81.6 | — | — | — | — | — | — | — | — | — | — | |
| 24-task CA-MTLBackbone=ROBERTa-LARGE, Multi-task learning strategy=CA-MTL2020.09 | 96.3 | 92 | 62.2 | 94.7 | 86.2 | — | 85.7 | — | — | — | 72.3 | 89 | 88.4 | — | — | 89.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| T5_LARGEArchitecture=Encoder-Decoder, Fine-tuning Protocol=jointly fine-tuned2022.08 | 96.3 | — | 61.2 | 94.8 | 87.2 | 89.9 | — | — | — | — | — | 89.9 | 89.6 | — | — | — | 89.9 | 89.9 | — | — | 87.35 | — | — | — | — | — | — | — | — | — | — | |
| T5-LargeParameters=770 million2019.10 | 96.3 | 89.9 | 61.2 | 94.8 | 87.2 | 89.9 | — | 92.4 | 89.2 | 73.9 | 89.9 | 89.9 | 89.6 | 85.6 | — | — | — | — | — | — | 86.4 | — | — | — | — | — | — | — | — | — | — | |
| AdapterHBackbone=RoBERTalarge, Batch size=32, Sequence length=128, Precision=FP16, #Param (%)=0.23, Peak Memory (GB)=6.05, Activation Memory (GB)=4.662023.06 | 96.3 | 87.7 | 66.3 | 94.7 | 72.9 | — | 86.4 | — | — | — | — | — | — | — | — | 91.5 | — | 91.5 | — | — | — | 90.3 | — | — | — | — | — | — | — | — | — | |
| AdapterBackbone=ROBERTa-large, # Params=0.8M2025.06 | 96.3 | 87.7 | 66.3 | 94.7 | 72.9 | — | 86.4 | — | — | — | 91.5 | — | — | — | — | 91.5 | — | — | — | 90.3 | — | — | — | — | — | — | — | — | — | — | — | |
| LoRA-XSBackbone=RoBERTa-Large, TP=0.06M2026.02 | 96.3 | 91.2 | — | 94.3 | 89.5 | 92.2 | 88.7 | — | — | — | — | — | — | — | — | — | — | — | 68.5 | — | — | — | — | — | — | — | — | — | — | — | — | |
| Batch PartitioningLM=RoBERTa2023.05 | 96.22 | 90.69 | 67.75 | 94.07 | 85.92 | 92.41 | 88.27 | 93.44 | 92.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 2.1 | — | — | — | — | — | — | |
| full AUTO-VBackbone=RoBERTa-large, Epsilon (ε)=infinity, Finetuning Protocol=non-DP2022.06 | 96.21 | — | — | 93.61 | — | — | — | — | — | — | 87.9 | 90.33 | 90.03 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Snorkel MeTaL2019.08 | 96.2 | 88.5 | 63.8 | 93.9 | 80.9 | 90.1 | 83.2 | 91.5 | 89.7 | 89.9 | 73.1 | 87.6 | 87.2 | 65.1 | 39.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AdapterHBackbone=RoBERTalarge, Batch size=32, Sequence length=128, Precision=FP16, #Param (%)=1.69, Peak Memory (GB)=6.18, Activation Memory (GB)=4.712023.06 | 96.2 | 88.7 | 66.5 | 94.7 | 83.4 | — | 87.8 | — | — | — | — | — | — | — | — | 91 | — | 92.1 | — | — | — | 89.9 | — | — | — | — | — | — | — | — | — | |
| LoRABackbone=RoBERTalarge, Batch size=32, Sequence length=128, Precision=FP16, #Param (%)=0.23, Peak Memory (GB)=6.11, Activation Memory (GB)=4.722023.06 | 96.2 | 90.2 | 68.2 | 94.8 | 85.2 | — | 88.6 | — | — | — | — | — | — | — | — | 92.3 | — | 91.6 | — | — | — | 90.6 | — | — | — | — | — | — | — | — | — | |
| LoRABackbone=ROBERTa-large, # Params=0.8M2025.06 | 96.2 | 93 | 68.1 | 94.6 | 85.2 | — | 88.9 | — | — | — | 91.6 | — | — | — | — | 92 | — | — | — | 90.7 | — | — | — | — | — | — | — | — | — | — | — | |
| Frozen knowledgeLM=RoBERTa2023.05 | 96.1 | 89.71 | 68.22 | 94.39 | 87.36 | 90.74 | 88.19 | 92.61 | 90.47 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 2.4 | — | — | — | — | — | — | |
| AdapterPBackbone=RoBERTalarge, Batch size=32, Sequence length=128, Precision=FP16, #Param (%)=0.85, Peak Memory (GB)=6.21, Activation Memory (GB)=4.782023.06 | 96.1 | 90.2 | 68.3 | 94.8 | 83.8 | — | 88.4 | — | — | — | — | — | — | — | — | 92.1 | — | 91.9 | — | — | — | 90.2 | — | — | — | — | — | — | — | — | — | |
| VeRABackbone=RoBERTa-Large, TP=0.06M2026.02 | 96.1 | 90.9 | — | 94.4 | 85.9 | 91.7 | 87.8 | — | — | — | — | — | — | — | — | — | — | — | 68 | — | — | — | — | — | — | — | — | — | — | — | — | |
| ELECTRA-Base++Train FLOPs=3.3e20, Params=110M, Finetuning=single-task2020.03 | 96 | 88.1 | — | 93.1 | 75.2 | — | 85.7 | — | — | — | 89.5 | — | — | — | — | 90.2 | — | — | 64.6 | 88.5 | — | — | — | — | — | — | — | — | — | — | — | |
| ELECTRA-BaseEvaluation Protocol=Single Task & Single Model2020.06 | 96 | — | 64.6 | 93.1 | 75.2 | — | 82.7 | — | — | — | — | — | — | 65.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| B6-3x2-3x2H768Evaluation Protocol=Single Task & Single Model, Layout=B6-3x2-3x2, Hidden Size=7682020.06 | 96 | — | 65.9 | 93.8 | 79.9 | — | 83.4 | — | — | — | — | — | — | 65.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ELECTRAModel type=Single model, Setting=BERT_BASE, Data augmentation=None2020.04 | 96 | 88.1 | 64.6 | 93.1 | 75.2 | — | 85.8 | — | — | — | 89.5 | — | — | — | — | 91 | — | — | — | 88.5 | — | — | — | — | — | — | — | — | — | — | — | |
| MPNetModel type=Single model, Setting=BERT_BASE, Data augmentation=None2020.04 | 96 | 89.1 | 64 | 93.1 | 81 | — | 86.5 | — | — | — | 89.9 | — | — | — | — | 90.7 | — | — | — | 88.5 | — | — | — | — | — | — | — | — | — | — | — | |
| LoRA-oBARBackbone=RoBERTa (355M), # para=0.8M2024.10 | 96 | 90.3 | 65.1 | 94.8 | 88.7 | — | — | — | — | — | 91.6 | — | — | — | — | 92.6 | — | — | — | 90.6 | 88.7 | — | — | — | — | — | — | — | — | — | — | |
| LoRA-nBARBackbone=RoBERTa (355M), # para=0.8M2024.10 | 96 | 90.3 | 65.6 | 94.7 | 89.2 | — | — | — | — | — | 91.6 | — | — | — | — | 92.6 | — | — | — | 90.8 | 88.9 | — | — | — | — | — | — | — | — | — | — | |
| FineGatesBackbone=RoBERTa-Large, TP=0.4M2026.02 | 96 | 91.2 | — | 94.1 | 90.2 | 92.3 | 89.2 | — | — | — | 89.4 | — | — | — | — | — | — | — | 71.4 | 89.1 | — | — | — | — | — | — | — | — | — | — | — | |
| SVFT^BBackbone=DeBERTaV3-base, d=2, #Params=0.28M2024.05 | 95.99 | 88.99 | 72.61 | 93.9 | 88.09 | — | 89.1 | — | — | — | 91.5 | — | — | — | — | 91.73 | — | — | — | 89.97 | — | — | — | — | — | — | — | — | — | — | — | |
| AD-DROPBackbone=RoBERTa-base, Protocol=Fine-tuned2022.10 | 95.9 | — | 58.5 | 93.4 | 76 | — | 84.76 | — | — | — | — | — | — | — | — | 89.3 | 87.9 | 89.5 | — | 87.6 | — | — | — | — | — | — | — | — | — | — | — | |
| MEFT1Backbone=RoBERTalarge, Batch size=32, Sequence length=128, Gradient=reversible, Precision=FP16, #Param (%)=0.23, Peak Memory (GB)=3.11, Activation Memory (GB)=1.732023.06 | 95.9 | 90.5 | 63.3 | 94.3 | 87.6 | — | 87.8 | — | — | — | — | — | — | — | — | 91.6 | — | 90.1 | — | — | — | 89.2 | — | — | — | — | — | — | — | — | — | |
| MEFT2Backbone=RoBERTalarge, Batch size=32, Sequence length=128, Gradient=reversible, Precision=FP32, #Param (%)=0.23, Peak Memory (GB)=3.75, Activation Memory (GB)=2.372023.06 | 95.9 | 90.5 | 64.4 | 94.3 | 88.2 | — | 87.9 | — | — | — | — | — | — | — | — | 92.1 | — | 89.4 | — | — | — | 88.4 | — | — | — | — | — | — | — | — | — | |
| LoRMA+Backbone=ROBERTa-large, # Params=0.8M2025.06 | 95.9 | 93 | 67.8 | 94.9 | 86.6 | — | 89 | — | — | — | 91.3 | — | — | — | — | 92.2 | — | — | — | 90.7 | — | — | — | — | — | — | — | — | — | — | — | |
| VeRABackbone=RoBERTa-Large, rank (r)=256, # Params=0.061M, Memory Cost=34.16G2024.07 | 95.83 | 87.72 | 63.66 | 94.11 | 83.03 | 91.12 | 85.91 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Full FTBackbone=BARTlarge, Batch size=100, Sequence length=128, Precision=FP16, #Param (%)=100, Peak Memory (GB)=12.75, Activation Memory (GB)=9.622023.06 | 95.8 | 89.2 | 59.3 | 94.3 | 77.6 | — | 85.2 | — | — | — | — | — | — | — | — | — | — | 89.5 | — | — | — | 90.8 | — | — | — | — | — | — | — | — | — | |
| LoRABackbone=RoBERTa (355M), # para=0.8M2024.10 | 95.8 | 89.6 | 64.8 | 94.7 | 88.2 | — | — | — | — | — | 91.4 | — | — | — | — | 92.4 | — | — | — | 90.6 | 88.4 | — | — | — | — | — | — | — | — | — | — | |
| C3ABackbone=RoBERTa-Large, b=1024/1, # Params=0.049M, Memory Cost=31.83G2024.07 | 95.78 | 88.02 | 66.59 | 94.22 | 82.89 | 91.86 | 86.56 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| C3ABackbone=RoBERTa-Large, b=1024/8, # Params=0.393M, Memory Cost=31.79G2024.07 | 95.78 | 88.09 | 67.18 | 94.26 | 84.62 | 91.81 | 86.96 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BOFTBackbone=RoBERTa-Large, block size (m)=2, # Params=0.442M, Memory Cost=34.98G2024.07 | 95.76 | 88.28 | 64.72 | 93.89 | 82.82 | 91.03 | 86.08 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FullBackbone=RoBERTa-Large, # Params=354M, Memory Cost=43.40G2024.07 | 95.75 | 88.35 | 64.87 | 92.4 | 84.48 | 91.65 | 86.25 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| RoBERTaLM=RoBERTa2023.05 | 95.64 | 90.49 | 66.84 | 93.37 | 86.28 | 91.83 | 87.86 | 93.07 | 91.95 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LoRABackbone=DeBERTaV3-base, rank=1, #Params=0.17M2024.05 | 95.64 | 86.43 | 69.13 | 94.18 | 87.36 | — | 88.23 | — | — | — | 91.43 | — | — | — | — | 91.52 | — | — | — | 90.12 | — | — | — | — | — | — | — | — | — | — | — | |
| Full-FTBackbone=DeBERTaV3-base, #Params=184M2024.05 | 95.63 | 89.46 | 69.19 | 94.03 | 83.75 | — | 88.25 | — | — | — | 92.4 | — | — | — | — | 91.6 | — | — | — | 89.9 | — | — | — | — | — | — | — | — | — | — | — | |
| MT-DNNModel Type=Single Model, #param=335M2019.11 | 95.6 | 86.7 | 62.5 | 93.1 | 75.5 | 88.3 | 82.7 | 90 | 87.7 | 72.4 | 89.6 | 86.7 | 86 | 65.1 | 40.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MT-DNNfine-tuning=true2019.01 | 95.6 | 91.1 | 62.5 | 93.1 | 81.4 | 89.5 | 82.7 | 88.2 | 88.8 | 89.6 | 72.7 | 86.7 | 86 | 65.1 | 40.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |