Natural Language Understanding on GLUE (dev)
97.36SST-2 (Acc)XDELECTRA-l
Evaluation results
| Method | Links | |||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| XDELECTRA-lmodel size=large2022.04 | 97.36 | 70.98 | — | — | 94.93 | — | 91.29 | 87.73 | — | — | 91.02 | — | — | 92.12 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 91.97 | — | — | — | — | — | — | — | — | — | |
| HTLM-R3F-Prompt# Params=400M, Fine-tuning=true2021.07 | 97.3 | 69.8 | 91.6 | 91.2 | 95.7 | — | 92.9 | 89.4 | — | — | — | — | — | 91.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FTModel=T5 Encoder, Bits=32, Trainable Params=1.2B2023.05 | 97.2 | 67.6 | — | — | 95.4 | — | — | — | — | 89.2 | 91.2 | — | — | — | — | — | — | — | 90.9 | 92.3 | 91.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 87.1 | — | — | — | — | — | — | — | |
| R4FBackbone=RoBERTa Large, Reporting Statistic=Best2020.08 | 97.1 | 70.6 | 90.1 | 90.8 | 95.1 | — | 92.5 | 88.8 | — | — | — | — | 89.9 | 90.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Back-translationBackbone=RoBERTa-large2020.09 | 97.1 | 69.4 | — | — | 95.3 | 93.5 | — | 91.7 | — | 90.1 | 91.1 | — | — | 90.9 | — | — | — | — | — | 92.8 | 92 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Feature CutoffBackbone=RoBERTa-large2020.09 | 97.1 | 71.1 | — | — | 95.2 | 93.4 | — | 90.9 | — | 90.1 | 90.9 | — | — | 90.9 | — | — | — | — | — | 92.4 | 92.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| HTLM-R3F# Params=400M, Fine-tuning=true2021.07 | 97.1 | 69.4 | 91.4 | 92.1 | 95.4 | — | 92.8 | 89.1 | — | — | — | — | — | 91.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| XLNetLearning Strategy=Single-task, Model Configuration=Single model2019.06 | 97 | 69 | 90.8 | 90.8 | 94.9 | — | 92.3 | 85.9 | — | — | — | — | — | 90.8 | 92.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| XLNetTrain FLOPs=3.9e21 (5.4x), Params=360M2020.03 | 97 | 69 | — | — | 94.9 | — | — | 85.9 | — | 89.1 | 90.8 | — | — | — | — | — | — | — | 90.8 | 92.2 | 92.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| XLNet-LargeEvaluation Protocol=Single Model2020.06 | 97 | 69 | — | — | 94.9 | — | — | 85.9 | — | 89.2 | 90.8 | — | — | — | — | — | — | — | — | 92.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| R3FBackbone=RoBERTa Large, Reporting Statistic=Best2020.08 | 97 | 71.2 | 91.1 | 91.3 | 95.3 | — | 92.4 | 88.5 | — | — | — | — | 89.9 | 91.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| XLNetModel size=large2020.06 | 97 | 69 | 90.8 | 90.8 | 94.9 | — | 92.3 | 85.9 | — | 89.15 | — | — | — | 90.8 | 92.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| XLNet-largeBackbone=XLNet-large, Protocol=Fine-tuning2021.06 | 97 | 69 | — | — | 94.9 | — | — | 85.9 | — | 89.15 | 90.8 | — | — | — | — | — | — | — | 90.8 | 92.5 | 92.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| CAMERO# of Models=4, Backbone=RoBERTa-large, Evaluation Protocol=Single-task fine-tuning, # Param. (million)=359.62022.04 | 97 | 70.3 | 91.1 | 90.9 | 95.3 | 94 | 92.5 | 87.7 | 92.6 | 89.8 | — | — | 90 | 91.7 | 92.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| RoBERTa-R3F# Params=330M, Fine-tuning=true2021.07 | 97 | 71.2 | 91.1 | 91.3 | 95.3 | — | 92.4 | 88.5 | — | — | — | — | — | 91.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SMART_RoBERTaArchitecture=RoBERTa_LARGE, Parameters=356 million2019.11 | 96.9 | 70.6 | 91.1 | 91.3 | 95.6 | 92.1 | 92.4 | 92 | 92.6 | — | — | — | 89.8 | 89.2 | 92.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ALBERTevaluation_protocol=single-task single-model, training_steps=1.5M2019.09 | 96.9 | 71.4 | — | — | 95.3 | — | 92.2 | 89.2 | — | — | 90.8 | — | — | 90.9 | 93 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ELECTRA-1.75MTrain FLOPs=3.1e21 (4.4x), Params=335M2020.03 | 96.9 | 69.1 | — | — | 95 | — | — | 88 | — | 89.5 | 90.9 | — | — | — | — | — | — | — | 90.8 | 92.6 | 92.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ELECTRA-LargeEvaluation Protocol=Single Model2020.06 | 96.9 | 69.1 | — | — | 95 | — | — | 88 | — | 89.5 | 90.9 | — | — | — | — | — | — | — | — | 92.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| STD++Backbone=RoBERTa Large, Reporting Statistic=Best2020.08 | 96.9 | 69.4 | 91 | — | 94.8 | — | 92.2 | 87.4 | — | — | — | — | — | 91.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SMARTBackbone=RoBERTa Large, Reporting Statistic=Best2020.08 | 96.9 | 70.6 | 91.1 | 91.3 | 95.6 | — | 92.4 | 92 | — | — | — | — | 89.8 | 89.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| STD++Backbone=RoBERTa Large, Reporting Statistic=Median, Number of runs=102020.08 | 96.9 | 68.4 | 90.8 | — | 92.5 | — | 92.1 | 87.4 | — | — | — | — | — | 89.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SMARTBackbone=RoBERTa-large2020.09 | 96.9 | 70.6 | — | — | 95.6 | 92.1 | — | 92 | — | 90 | 91.1 | — | — | 89.2 | — | — | — | — | — | 92.8 | 92.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Token CutoffBackbone=RoBERTa-large2020.09 | 96.9 | 70 | — | — | 95.3 | 93.2 | — | 90.6 | — | 90 | 91 | — | — | 90.9 | — | — | — | — | — | 92.5 | 92.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Span CutoffBackbone=RoBERTa-large2020.09 | 96.9 | 71.5 | — | — | 95.3 | 93.8 | — | 91 | — | 90.3 | 91.1 | — | — | 91.4 | — | — | — | — | — | 92.8 | 92.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ELECTRAModel size=large2020.06 | 96.9 | 69.1 | 90.9 | — | 95 | — | 92.4 | 88 | — | 89.46 | — | — | — | 90.8 | 92.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ELECTRA-largeBackbone=ELECTRA-large, Protocol=Fine-tuning2021.06 | 96.9 | 69.1 | — | — | 95 | — | — | 88 | — | 89.46 | 90.9 | — | — | — | — | — | — | — | 90.8 | 92.6 | 92.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| RoBERTa-large + R-DropBackbone=RoBERTa-large, Protocol=Fine-tuning, Regularization=R-Drop2021.06 | 96.9 | 70 | — | — | 95.2 | — | — | 88.4 | — | 89.73 | 90.9 | — | — | — | — | — | — | — | 91.4 | 92.5 | 92.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ELECTRA-lmodel size=large2022.04 | 96.9 | 69.27 | — | — | 94.78 | — | 91.34 | 86.64 | — | — | 90.99 | — | — | 91.53 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 91.88 | — | — | — | — | — | — | — | — | — | |
| PreQuantModel=T5 Encoder, Bits=4, Trainable Params=11.80M2023.05 | 96.9 | 66.4 | — | — | 95.1 | — | — | — | — | 88.5 | 90.7 | — | — | — | — | — | — | — | 90 | 91.6 | 92 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 85.1 | — | — | — | — | — | — | — | |
| ELECTRA_LARGEArchitecture=Encoder-Only2022.08 | 96.9 | — | 90.9 | — | 95 | — | — | 88 | — | 89.46 | — | — | — | — | — | — | — | — | 90.8 | 92.6 | 92.4 | — | — | 69.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DeBERTaV3_LARGEArchitecture=Encoder-Only2022.08 | 96.9 | — | 91.8 | 91.9 | 96 | — | — | 92.7 | — | 91.37 | — | — | — | — | — | — | — | — | 92.2 | 93 | 93 | — | — | 75.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| HTLM# Params=400M, Fine-tuning=true2021.07 | 96.9 | 64.3 | 90.3 | 91.4 | 95.1 | — | 92.6 | 87.1 | — | — | — | — | — | 90.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ALBERTevaluation_protocol=single-task single-model, training_steps=1M2019.09 | 96.8 | 68.7 | — | — | 95.2 | — | 92 | 88.1 | — | — | 90.4 | — | — | 90.2 | 92.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| B10-10-10H1024Evaluation Protocol=Single Model, Layout=B10-10-10, Hidden Size=10242020.06 | 96.8 | 72.4 | — | — | 95.1 | — | — | 89.5 | — | 90 | — | — | — | — | — | — | — | — | — | 92.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| B8-8-8H1024Evaluation Protocol=Single Model, Layout=B8-8-8, Hidden Size=10242020.06 | 96.8 | 71.3 | — | — | 94.7 | — | — | 89.2 | — | 89.7 | — | — | — | — | — | — | — | — | — | 91.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| R4FBackbone=RoBERTa Large, Reporting Statistic=Median, Number of runs=102020.08 | 96.8 | 70.1 | 90 | 90.6 | 94.8 | — | 91.8 | 88.3 | — | — | — | — | 88.2 | 90.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FreeLBBackbone=RoBERTa-large2020.09 | 96.8 | 71.1 | — | — | 95 | — | — | 88.1 | — | 89.8 | 90.6 | — | — | 91.4 | — | — | — | — | — | 92.7 | 92.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DeBERTaModel size=large2020.06 | 96.8 | 70.5 | 91.1 | 91.1 | 95.3 | — | 92.3 | 88.3 | — | 90 | — | — | — | 91.9 | 92.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DeBERTa_LARGEArchitecture=Encoder-Only2022.08 | 96.8 | — | 91.1 | 91.1 | 95.3 | — | — | 88.3 | — | 90 | — | — | — | — | — | — | — | — | 91.9 | 92.8 | 92.3 | — | — | 70.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SuperICLLLM=GPT-3.5, Plug-in model=RoBERTa-large2023.05 | 96.79 | 64.57 | 89.31 | 89.61 | 94.16 | — | 92.14 | 87.73 | — | 89.9 | — | — | — | 86.03 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FreeLBArchitecture=RoBERTa_LARGE, Parameters=356 million2019.11 | 96.7 | 71.1 | 90.6 | — | 95 | 91.4 | 92.6 | 88.1 | — | — | — | — | — | — | 92.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FreeLBBackbone=RoBERTa Large, Reporting Statistic=Best2020.08 | 96.7 | 71.1 | 90.6 | — | 95 | — | 92.6 | 88.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MPNetModel Scale=Large, Setting=BERT_LARGE, Initialization=RoBERTa_LARGE2020.04 | 96.7 | 69.1 | — | — | 94.9 | — | — | 88 | — | 89.4 | 90.5 | — | — | — | — | — | — | — | 91.4 | 92.4 | 92.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| QATModel=T5 Encoder, Bits=4, Trainable Params=1.2B2023.05 | 96.7 | 66.5 | — | — | 95.3 | — | — | — | — | 88.6 | 90.4 | — | — | — | — | — | — | — | 90.2 | 91.6 | 91.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 86.6 | — | — | — | — | — | — | — | |
| ALUM_ROBERTA-LARGEArchitecture=RoBERTa-large, Training=Adversarial Training (ALUM)2020.04 | 96.6 | 68.2 | 90.9 | — | 95.1 | — | — | 87.3 | — | — | — | — | — | — | — | — | — | — | 91.1 | 92.1 | 92.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SMARTBackbone=RoBERTa Large, Reporting Statistic=Median, Number of runs=102020.08 | 96.6 | 69.4 | 90.85 | 91.1 | 94.8 | — | 91.7 | 89.5 | — | — | — | — | 88.2 | 83.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ALUMBackbone=RoBERTa-large2020.09 | 96.6 | 68.2 | — | — | 95.1 | — | — | 87.3 | — | 89.2 | 90.9 | — | — | 91.1 | — | — | — | — | — | 92.1 | 92.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Super (11B)Architecture=T5-11B, FLOPs=87e112022.03 | 96.6 | 69.1 | — | — | 95.9 | — | — | 92.4 | — | — | 91.7 | — | — | 91.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| E-LANGArchitecture=T5, Super model=T5-11B, Swift model=T5-Large, Avg. Speed-up (FLOPs)=3.3X, Avg. Speed-up (time)=1.8X2022.03 | 96.6 | 69.5 | — | — | 96 | — | — | 92.4 | — | — | 91.7 | — | — | 92.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BART-Large# Params=400M, Fine-tuning=true2021.07 | 96.6 | 62.8 | 89.9 | 90.1 | 94.9 | — | 92.5 | 87 | — | — | — | — | — | 90.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Vanilla (RoBERTa-large)# of Models=4, Backbone=RoBERTa-large, Evaluation Protocol=Single-task fine-tuning, # Param. (million)=1425.62022.04 | 96.5 | 68.2 | 90.8 | 90.5 | 94.7 | 93.6 | 92.4 | 86.2 | 92.5 | 89 | — | — | 89.8 | 91.2 | 92.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Z-Code++Architecture=Encoder-Decoder2022.08 | 96.5 | — | 91.4 | 91.4 | 95.2 | — | — | 92.1 | — | 92.19 | — | — | — | — | — | — | — | — | 91.2 | 92.5 | 92.4 | — | — | 86.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| RoBERTa-LargeMode=Fine-tuned2023.05 | 96.44 | 64.55 | 88.68 | 89.47 | 94.07 | — | 92.11 | 87 | — | 88.68 | — | — | — | 83.09 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| RoBERTaArchitecture=RoBERTa_LARGE, Parameters=356 million2019.11 | 96.4 | 68 | 90.2 | — | 94.7 | 90.9 | 92.2 | 86.6 | — | — | — | — | — | — | 92.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PGDArchitecture=RoBERTa_LARGE, Parameters=356 million2019.11 | 96.4 | 69.7 | 90.5 | — | 94.9 | 90.9 | 92.5 | 87.4 | — | — | — | — | — | — | 92.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| RoBERTaMode=Single-task single models, Architecture=24-layer2019.07 | 96.4 | 68 | 90.2 | 90.2 | 94.7 | — | 92.2 | 86.6 | — | — | — | 91.3 | — | 90.9 | 92.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ROBERTaLearning Strategy=Single-task, Model Configuration=Single model2019.06 | 96.4 | 68 | 90.2 | 90.2 | 94.7 | — | 92.2 | 86.6 | — | — | — | — | — | 90.9 | 92.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| RoBERTa-largeevaluation_protocol=single-task single-model2019.09 | 96.4 | 68 | — | — | 94.7 | — | 92.2 | 86.6 | — | — | 90.2 | — | — | 90.9 | 92.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| RoBERTa-500KTrain FLOPs=3.2e21 (4.5x), Params=356M2020.03 | 96.4 | 68 | — | — | 94.7 | — | — | 86.6 | — | 88.9 | 90.2 | — | — | — | — | — | — | — | 90.9 | 92.1 | 92.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ROBERTA_LARGEArchitecture=RoBERTa-large2020.04 | 96.4 | 67.8 | 90.2 | — | 94.7 | — | — | 86.6 | — | — | — | — | — | — | — | — | — | — | 90.9 | 92.4 | 92.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| RoBERTa-LargeEvaluation Protocol=Single Model2020.06 | 96.4 | 68 | — | — | 94.7 | — | — | 86.6 | — | 88.9 | 90.2 | — | — | — | — | — | — | — | — | 92.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| STDBackbone=RoBERTa Large, Reporting Statistic=Best2020.08 | 96.4 | 68 | 90.2 | — | 94.7 | — | 92.2 | 86.6 | — | — | — | — | — | 89.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| STDBackbone=RoBERTa Large, Reporting Statistic=Median, Number of runs=102020.08 | 96.4 | 66.2 | 90.2 | — | 92.1 | — | 91.9 | 86.6 | — | — | — | — | — | 84.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ROBERTa-largeBackbone=RoBERTa-large2020.09 | 96.4 | 68 | — | — | 94.7 | 90.9 | — | 86.6 | — | — | 90.2 | — | — | — | — | — | — | — | — | 92.4 | 92.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PGDBackbone=RoBERTa-large2020.09 | 96.4 | 69.7 | — | — | 94.9 | — | — | 87.4 | — | 89.3 | 90.5 | — | — | 90.9 | — | — | — | — | — | 92.4 | 92.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| RoBERTaModel Scale=Large, Setting=BERT_LARGE2020.04 | 96.4 | 68 | — | — | 94.7 | — | — | 86.6 | — | 88.9 | 90.2 | — | — | — | — | — | — | — | 90.9 | 92.4 | 92.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| RoBERTaModel size=large2020.06 | 96.4 | 68 | 90.2 | 90.2 | 93.9 | — | 92.2 | 86.6 | — | 88.82 | — | — | — | 90.9 | 92.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| RoBERTa-largeBackbone=RoBERTa-large, Protocol=Fine-tuning2021.06 | 96.4 | 68 | — | — | 94.7 | — | — | 86.6 | — | 88.93 | 90.2 | — | — | — | — | — | — | — | 90.9 | 92.4 | 92.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Single (RoBERTa-large)# of Models=1, Backbone=RoBERTa-large, Evaluation Protocol=Single-task fine-tuning, # Param. (million)=356.42022.04 | 96.4 | 68 | 90.2 | 90.2 | 94.7 | — | 92.2 | 86.6 | 92.4 | 88.9 | — | — | — | 90.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FTModel=RoBERTa-large, Bits=32, Trainable Params=302M2023.05 | 96.4 | 68 | — | — | 94.7 | — | — | — | — | 88.9 | 90.2 | — | — | — | — | — | — | — | 90.9 | 92.4 | 92.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 86.6 | — | — | — | — | — | — | — | |
| FinetuningBackbone=RoBERTa-large, # Params=355M2023.05 | 96.4 | 68 | — | — | 94.7 | — | 92.2 | 86.6 | — | 88.9 | 90.2 | — | — | — | — | 92.4 | 90.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| RoBERTa_LARGEArchitecture=Encoder-Only2022.08 | 96.4 | — | 90.2 | 90.2 | 93.9 | — | — | 86.6 | — | 88.82 | — | — | — | — | — | — | — | — | 90.9 | 92.4 | 92.2 | — | — | 68 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| RoBERTa# Params=330M, Fine-tuning=true2021.07 | 96.4 | 68 | 90.2 | — | 94.7 | — | 92.2 | 86.6 | — | — | — | — | — | 89.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ADAPAD#Params=0.34M, Backbone=DeBERTaV3-base, rmax=42026.05 | 96.33 | 72.26 | — | — | 94.36 | — | 91.55 | 86.28 | — | 89.34 | 90.37 | — | — | 91.67 | 91.86 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| B6-6-6H768Evaluation Protocol=Single Model, Layout=B6-6-6, Hidden Size=7682020.06 | 96.3 | 70.1 | — | — | 93.7 | — | — | 83.4 | — | 88.3 | — | — | — | — | — | — | — | — | — | 91.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AdapterBackbone=RoBERTa-large, # Params=3M2023.05 | 96.3 | 67.4 | — | — | 94.7 | — | 88.5 | 83.4 | — | 88.3 | 90.4 | — | — | — | — | 92.5 | 92.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Z-Code++Architecture=Encoder-Only2022.08 | 96.3 | — | 91.7 | 91.5 | 95.8 | — | — | 92.4 | — | 91.23 | — | — | — | — | — | — | — | — | 92.4 | 93.1 | 92.8 | — | — | 75.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| RoAd1Model=RoBERTa-large, #Params.=0.06%2024.08 | 96.3 | 66.1 | — | — | 94.4 | — | 91 | 89.2 | — | 88.7 | 89.7 | — | — | — | — | — | — | — | 91 | 91.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| T5-Large# Params=770M, Fine-tuning=true2021.07 | 96.3 | 61.2 | 89.9 | 89.6 | 94.8 | — | 89.9 | 87.2 | — | — | — | — | — | 89.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| IncreLoRA#Params=0.34M, Backbone=DeBERTaV3-base, reproduced=true2026.05 | 96.22 | 71.85 | — | — | 93.5 | — | 91.94 | 85.92 | — | 88.99 | 90.59 | — | — | 90.2 | 91.68 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| R3FBackbone=RoBERTa Large, Reporting Statistic=Median, Number of runs=102020.08 | 96.2 | 70.6 | 91.1 | 91.1 | 95.1 | — | 92.1 | 88.4 | — | — | — | — | 88.4 | 91.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LoRABackbone=RoBERTa-large, # Params=800K2023.05 | 96.2 | 68.2 | — | — | 94.9 | — | 88.8 | 87.4 | — | 89 | 90.8 | — | — | — | — | 92.6 | 93.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LoReFTModel=RoBERTa-large, #Params.=0.01%2024.08 | 96.2 | 68 | — | — | 94.1 | — | 88.5 | 87.5 | — | 88.2 | 89.2 | — | — | — | — | — | — | — | 90.1 | 91.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LoRARank (r)=8, Seeds=32026.05 | 96.14 | — | — | — | — | — | — | — | — | 88.67 | 90.31 | — | — | — | — | — | — | — | 92.22 | 91.25 | 88.99 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 85.92 | — | — | 70.46 | — | 94.09 | — | — | |
| FreeATArchitecture=RoBERTa_LARGE, Parameters=356 million2019.11 | 96.1 | 68.8 | 90 | — | 94.7 | 90.7 | 92.5 | 86.7 | — | — | — | — | — | — | 92.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FreeATBackbone=RoBERTa-large2020.09 | 96.1 | 68.8 | — | — | 94.7 | — | — | 86.7 | — | 89 | 90 | — | — | 90.7 | — | — | — | — | — | 92.4 | 92.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Adapter FNNModel=RoBERTa-large, #Params.=0.23%2024.08 | 96.1 | 64.4 | — | — | 94.3 | — | 91.3 | 84.8 | — | 87.7 | 90.3 | — | — | — | — | — | — | — | 90.5 | 90.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| RoAd1(fc1)Model=RoBERTa-large, #Params.=0.03%2024.08 | 96.1 | 68.1 | — | — | 94.5 | — | 90.2 | 88.7 | — | 88.8 | 89.6 | — | — | — | — | — | — | — | 91.5 | 91.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ERNIE 2.0Model scale=Large, Median of five runs=true2019.07 | 96 | 65.4 | 89.1 | — | 94.3 | — | 92.5 | 85.2 | — | 83.6 | — | — | — | 89.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ELECTRA-400KTrain FLOPs=7.1e20 (1x), Params=335M2020.03 | 96 | 69.3 | — | — | 94.5 | — | — | 86.8 | — | 89 | 90.5 | — | — | — | — | — | — | — | 90.6 | 92.1 | 92.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| WARP20Backbone=RoBERTa-large, # Params=25K2023.05 | 96 | 60.6 | — | — | 93.5 | — | 84.5 | 75.8 | — | 84.8 | 88.2 | — | — | — | — | 88.6 | 90.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Full FTModel=RoBERTa-large, #Params.=100.00%2024.08 | 96 | 68.2 | — | — | 93.8 | — | 91.5 | 85.8 | — | 88.6 | 88.8 | — | — | — | — | — | — | — | 91.7 | 92.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LORAModel=RoBERTa-large, #Params.=0.23%2024.08 | 96 | 65.5 | — | — | 94.7 | — | 90.7 | 86.3 | — | 88.1 | 90.1 | — | — | — | — | — | — | — | 89.8 | 91.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| REDModel=RoBERTa-large, #Params.=0.01%2024.08 | 96 | 68.1 | — | — | 93.5 | — | 88.8 | 86.2 | — | 88 | 89.5 | — | — | — | — | — | — | — | 90.3 | 91.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| EVARank (r)=8, Seeds=32026.05 | 95.95 | — | — | — | — | — | — | — | — | 88.57 | 90.11 | — | — | — | — | — | — | — | 91.96 | 91.54 | 89.02 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 85.32 | — | — | 70.46 | — | 94.24 | — | — | |
| FIM-LoRARank (r)=8, Seeds=32026.05 | 95.95 | — | — | — | — | — | — | — | — | 88.6 | 90.25 | — | — | — | — | — | — | — | 92.11 | 91.33 | 89.06 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 85.44 | — | — | 70.44 | — | 94.24 | — | — | |
| BERTTrain FLOPs=7.1e20 (1x), Params=335M, Note=Ours2020.03 | 95.9 | 67 | — | — | 93.5 | — | — | 79.5 | — | 87.2 | 89.6 | — | — | — | — | — | — | — | 89.1 | 91.2 | 91.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LoRABackbone=RoBERTa-large, # Params=101K, rank=12023.05 | 95.9 | 63.3 | — | — | 94.3 | — | 87.1 | 85.6 | — | 87.6 | 90 | — | — | — | — | 91.9 | 92.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AdapterBackbone=RoBERTa-large, # Params=150K, m=12023.05 | 95.9 | 68 | — | — | 94.7 | — | 88 | 85.6 | — | 88.4 | 90.4 | — | — | — | — | 92.1 | 92.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |