Natural Language Understanding on GLUE (test dev)
93.45MRPC AccuracySL-SAM
Evaluation results
| Method | Links | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| SL-SAMBackbone=RoBERTa, Active Ratio=0.441x, Training Mode=Fine-tuning2026.02 | 93.45 | 95.41 | 91.63 | 87.55 | 92.64 | — | 79.78 | 63.36 | 90.38 | 86.78 | — | — | |
| ESAMBackbone=RoBERTa, Active Ratio=1.5x, Training Mode=Fine-tuning2026.02 | 92.88 | 94.5 | 91.97 | 87.52 | 92.9 | — | 80.51 | 63.41 | 90.96 | 86.83 | — | — | |
| SSAM-FBackbone=RoBERTa, Active Ratio=2x, Training Mode=Fine-tuning2026.02 | 92.73 | 95.53 | 92.16 | 87.59 | 93.01 | — | 80.14 | 61.32 | 91.11 | 86.7 | — | — | |
| AdaSAMBackbone=RoBERTa, Active Ratio=2x, Training Mode=Fine-tuning2026.02 | 92.5 | 95.41 | 92.12 | 87.46 | 92.99 | — | 80.87 | 63.08 | 90.92 | 86.92 | — | — | |
| RSTBackbone=RoBERTa, Active Ratio=1.5x, Training Mode=Fine-tuning2026.02 | 92.39 | 95.07 | 91.89 | 87.57 | 93.25 | — | 79.42 | 62.38 | 90.97 | 86.62 | — | — | |
| Our Full FTTrainable Parameters=100%, Backbone=RoBERTa-Large2023.05 | 91.7 | 96.4 | 92.3 | 90.1 | 94.8 | — | 88.1 | 69 | 91.9 | 89.3 | — | — | |
| AdamWBackbone=RoBERTa, Active Ratio=1x, Training Mode=Fine-tuning2026.02 | 91.68 | 94.04 | 91.9 | 87.43 | 92.75 | — | 77.98 | 61.64 | 90.91 | 86.04 | — | — | |
| PaFiTrainable Parameters=0.5%, Backbone=RoBERTa-Large2023.05 | 91.4 | 96.7 | 90.3 | 90.2 | 94.6 | — | 88.8 | 70.2 | 91.9 | 89.3 | — | — | |
| HiddenKeyCompensation=KL2024.02 | 91.2 | 96.44 | — | — | — | — | 88.1 | — | 92.3 | 92.01 | — | — | |
| DropKeyPattern=element2024.02 | 90.93 | 96.22 | — | — | — | — | 87 | — | 92.21 | 91.59 | — | — | |
| DropKeyPattern=column2024.02 | 90.93 | 96.22 | — | — | — | — | 87.36 | — | 92.25 | 91.69 | — | — | |
| Full FTTrainable Parameters=100%, Backbone=RoBERTa-Large2023.05 | 90.9 | 96.4 | 92.2 | 90.2 | 94.7 | — | 86.6 | 68 | 92.4 | 88.9 | — | — | |
| Full FinetuningPosition=Full Finetuning2024.02 | 90.9 | 96.4 | — | — | — | — | 86.6 | — | 92.4 | 91.58 | — | — | |
| HiddenKeyPattern=element2024.02 | 90.9 | 96.22 | — | — | — | — | 87.7 | — | 92.28 | 91.78 | — | — | |
| HiddenKeyCompensation=JS2024.02 | 90.9 | 96.22 | — | — | — | — | 87.7 | — | 92.24 | 91.77 | — | — | |
| HiddenKeyCompensation=input2024.02 | 90.7 | 96.33 | — | — | — | — | 88.5 | — | 92.11 | 91.16 | — | — | |
| HiddenKeyCompensation=output2024.02 | 90.7 | 96.22 | — | — | — | — | 87.7 | — | 92.19 | 90.95 | — | — | |
| HiddenCutPattern=element2024.02 | 90.69 | 96.1 | — | — | — | — | 87 | — | 91.94 | 91.43 | — | — | |
| HiddenCutPattern=span2024.02 | 90.69 | 96.1 | — | — | — | — | 86.64 | — | 92.05 | 91.37 | — | — | |
| DropKeyPattern=span2024.02 | 90.69 | 96.22 | — | — | — | — | 86.28 | — | 92.21 | 91.35 | — | — | |
| Pfeiffer AdapterTrainable Parameters=0.8%, Backbone=RoBERTa-Large2023.05 | 90.2 | 96.1 | 91.8 | 90.2 | 94.8 | — | 83.8 | 68.3 | 92.1 | 88.4 | — | — | |
| LoRATrainable Parameters=0.2%, Backbone=RoBERTa-Large2023.05 | 90.2 | 96.2 | 91.6 | 90.6 | 94.8 | — | 85.2 | 68.2 | 92.3 | 88.6 | — | — | |
| Diff PruningTrainable Parameters=0.5%, Backbone=RoBERTa-Large2023.05 | 90.2 | 96.4 | 90.3 | 90.3 | 94.6 | — | 84.5 | 65.1 | 92 | 87.9 | — | — | |
| HiddenCutPattern=column2024.02 | 90.2 | 96.22 | — | — | — | — | 86.64 | — | 91.96 | 91.26 | — | — | |
| DropAttentionPattern=element2024.02 | 90.2 | 95.76 | — | — | — | — | 85.56 | — | 92.03 | 90.89 | — | — | |
| DropAttentionPattern=column2024.02 | 90.2 | 95.87 | — | — | — | — | 85.56 | — | 92.11 | 90.94 | — | — | |
| ROBERTABASEsize=Base2025.07 | 90 | 96 | 84 | 84 | 92 | — | — | — | — | — | — | — | |
| BaselinePosition=Baseline2024.02 | 89.95 | 95.99 | — | — | — | — | 84.48 | — | 91.96 | 90.6 | — | — | |
| DropAttentionPattern=span2024.02 | 89.95 | 96.1 | — | — | — | — | 86.28 | — | 92.21 | 91.14 | — | — | |
| Pfeiffer AdapterTrainable Parameters=0.2%, Backbone=RoBERTa-Large2023.05 | 89.7 | 96.6 | 91.9 | 90.5 | 94.7 | — | 80.1 | 67.8 | 91.9 | 87.9 | — | — | |
| BERTBASEsize=Base2025.07 | 88.9 | 88 | 71.2 | 84.6 | 90.5 | — | — | — | — | — | — | — | |
| AdapterTrainable Parameters=1.7%, Backbone=RoBERTa-Large2023.05 | 88.7 | 96.2 | 92.1 | 89.9 | 94.8 | — | 83.4 | 66.5 | 91 | 87.8 | — | — | |
| FISH MaskTrainable Parameters=0.5%, Backbone=RoBERTa-Large2023.05 | 88.7 | 96.1 | 89.8 | 90.2 | 94.1 | — | 86.3 | 66.3 | 92.5 | 88 | — | — | |
| RoBERTabase + VAWI-LBSBase Model=RoBERTabase, Strategy=Learning-based strategy (LBS)2022.12 | 88.5 | 91.6 | 87.9 | 82.4 | 90.6 | — | — | — | 88.3 | 88.21 | — | — | |
| RoBERTabase + VAWI-VABSBase Model=RoBERTabase, Strategy=Visually-enhanced attention based strategy (VABS)2022.12 | 88.3 | 91.7 | 87.9 | 82.6 | 89.1 | — | — | — | 88.1 | 87.95 | — | — | |
| RoBERTabase + VAWI-SBSBase Model=RoBERTabase, Strategy=Syntax-based strategy (SBS)2022.12 | 88.2 | 91.4 | 87.7 | 82.2 | 89.4 | — | — | — | 87.7 | 87.76 | — | — | |
| RoBERTabase + iACEBase Model=RoBERTabase, Modification=iACE2022.12 | 87.7 | 91.6 | 87.9 | 82.6 | 89.1 | — | — | — | 86.9 | 87.63 | — | — | |
| AdapterTrainable Parameters=0.8%, Backbone=RoBERTa-Large2023.05 | 87.7 | 96.3 | 91.7 | 90.3 | 94.7 | — | 72.9 | 66.3 | 91.5 | 86.4 | — | — | |
| BERT-base + EDBackbone=BERT-base2026.05 | 87.66 | — | — | — | — | — | 71.12 | 62.45 | — | — | — | — | |
| RoBERTabase + VOKENBase Model=RoBERTabase, Modification=VOKEN2022.12 | 87 | 90.5 | 87.8 | 81 | 89.2 | — | — | — | 86.9 | 87.06 | — | — | |
| MaBERTPre-training budget=100% steps2026.03 | 86.9 | 93.3 | 87.9 | — | 89.3 | — | 65.4 | 67.6 | — | — | 83.5 | 83.7 | |
| BERT-baseBackbone=BERT-base2026.05 | 86.74 | — | — | — | — | — | 70.28 | 62.31 | — | — | — | — | |
| ALBERT-BASEBackbone=ALBERT-base2021.10 | 86.5 | 92.4 | 90.8 | 85.1 | 90.9 | — | 75.8 | 54.5 | 90.3 | — | — | — | |
| BigBirdPre-training budget=100% steps2026.03 | 86.4 | 92.6 | 85.7 | — | 88.1 | — | 62.4 | 52.8 | — | — | 83.1 | 83.2 | |
| FP BERTPrecision=Full-precision2024.05 | 86.3 | 93.2 | 91.4 | 84.9 | 92.1 | — | 72.2 | 59.7 | 90.1 | 83.9 | — | — | |
| LongformerPre-training budget=100% steps2026.03 | 86.3 | 92.4 | 85.8 | — | 88.2 | — | 62.6 | 53.4 | — | — | 83 | 83.1 | |
| BERT-base + MixupBackbone=BERT-base2026.05 | 86.3 | — | — | — | — | — | 70.28 | 61.04 | — | — | — | — | |
| DeBERTaPre-training budget=100% steps2026.03 | 86.2 | 93.4 | 86.8 | — | 88.6 | — | 64.8 | 61.7 | — | — | 83.8 | 84.2 | |
| MobileBERT#Pr (Fine-tuned parameters in Millions)=25.3, #To (Total parameters in Millions)=25.32021.06 | 86 | — | — | — | — | 56.2 | 63.5 | — | — | — | — | — | |
| OmniVec22025.07 | 85.8 | 95.6 | 82.2 | 87.9 | 84.2 | — | — | — | — | — | — | — | |
| BERTbase + iACEBase Model=BERTbase, Modification=iACE2022.12 | 85.8 | 91.7 | 89.1 | 82.8 | 88.6 | — | — | — | 86.6 | 87.43 | — | — | |
| BERTbase + VAWI-VABSBase Model=BERTbase, Strategy=Visually-enhanced attention based strategy (VABS)2022.12 | 85.8 | 92.7 | 89.5 | 82.7 | 88.9 | — | — | — | 87.2 | 87.8 | — | — | |
| BERTbase + VAWI-LBSBase Model=BERTbase, Strategy=Learning-based strategy (LBS)2022.12 | 85.6 | 92.4 | 89.7 | 83 | 89.1 | — | — | — | 86.9 | 87.78 | — | — | |
| BERT#Pr (Fine-tuned parameters in Millions)=110, #To (Total parameters in Millions)=1102021.06 | 85.5 | — | — | — | — | 56.3 | 70 | — | — | — | — | — | |
| BERTbase + VAWI-SBSBase Model=BERTbase, Strategy=Syntax-based strategy (SBS)2022.12 | 85.5 | 92.9 | 89.6 | 82.2 | 88.4 | — | — | — | 86.9 | 87.58 | — | — | |
| ALBERTPre-training budget=100% steps2026.03 | 85.5 | 92 | 85.7 | — | 88 | — | 61.8 | 50.3 | — | — | 82.9 | 83.2 | |
| MPOPMBackbone=MobileBERT, #Pr (Fine-tuned parameters in Millions)=4.4, #To (Total parameters in Millions)=15.42021.06 | 85.3 | — | — | — | — | 56.2 | 65.7 | — | — | — | — | — | |
| BERTPre-training budget=100% steps2026.03 | 85.3 | 91.2 | 85.6 | — | 87.6 | — | 61.8 | 52.2 | — | — | 82.6 | 82.9 | |
| BILSTM+ELMo+Attn2025.07 | 84.9 | 90.4 | 64.8 | 76.4 | 79.8 | — | — | — | — | — | — | — | |
| MPOPBBackbone=BERT, #Pr (Fine-tuned parameters in Millions)=7.7, #To (Total parameters in Millions)=70.42021.06 | 84.3 | — | — | — | — | 56.3 | 70.8 | — | — | — | — | — | |
| MPOPDBackbone=DistilBERT, #Pr (Fine-tuned parameters in Millions)=4.0, #To (Total parameters in Millions)=43.42021.06 | 84.3 | — | — | — | — | 56.3 | 61.7 | — | — | — | — | — | |
| DistilBERT#Pr (Fine-tuned parameters in Millions)=66, #To (Total parameters in Millions)=662021.06 | 84.1 | — | — | — | — | 56.3 | 61.4 | — | — | — | — | — | |
| BERTbase + VOKENBase Model=BERTbase, Modification=VOKEN2022.12 | 83.5 | 92.2 | 88.6 | 82.6 | 88.6 | — | — | — | 86 | 86.83 | — | — | |
| OpenAI GPT2025.07 | 82.3 | 91.3 | 70.3 | 82.1 | 87.4 | — | — | — | — | — | — | — | |
| Meta-Transformer-B16Tbackbone=B16T2025.07 | 81.8 | 81.3 | 78 | 70 | 60.3 | — | — | — | — | — | — | — | |
| BERTbaseBase Model=BERTbase, Modification=Fine-tune (None)2022.12 | 81.7 | 89.3 | 87.2 | 79.4 | 87.9 | — | — | — | 84.4 | 84.98 | — | — | |
| RoBERTabaseBase Model=RoBERTabase, Modification=Fine-tune (None)2022.12 | 81.4 | 89.2 | 86.2 | 79 | 87.5 | — | — | — | 85.4 | 84.78 | — | — | |
| TENCBase Model=Mirror-RoBERTa-base-drophead, Encoding=cross-encoder2021.09 | 81.38 | — | 82.9 | — | 82.52 | — | — | — | — | 82.27 | — | — | |
| BITBitwidth=1-bit2024.05 | 79.7 | 87.7 | 82.9 | 77.1 | 85.7 | — | 58.8 | 25.1 | 71.1 | 71 | — | — | |
| TENCBase Model=Mirror-RoBERTa-base-drophead, Encoding=bi-encoder2021.09 | 79.63 | — | 82.04 | — | 82.12 | — | — | — | — | 81.26 | — | — | |
| BIT (Reprod.)Bitwidth=1-bit, Reproduced=true2024.05 | 78.9 | 87.5 | 87.2 | 76.8 | 85.6 | — | 58.8 | 24.1 | 70.5 | 69.7 | — | — | |
| TENCBase Model=Mirror-RoBERTa-base, Encoding=cross-encoder2021.09 | 78.4 | — | 82 | — | 79.87 | — | — | — | — | 80.09 | — | — | |
| BOLDBitwidth=1-bit2024.05 | 78.4 | 88.7 | 85.9 | 75.6 | 84.1 | — | 58.8 | 27.1 | 68.7 | 70.9 | — | — | |
| BLIPBase Model=BLIP, Modification=Fine-tune (None)2022.12 | 77.8 | 76.3 | 78.8 | 72.5 | 77.4 | — | — | — | 76.4 | 76.53 | — | — | |
| Mirror-RoBERTa-base-dropheadBase Model=Mirror-RoBERTa-base-drophead2021.09 | 77.18 | — | 78.36 | — | 75.56 | — | — | — | — | 77.03 | — | — | |
| TENCBase Model=Mirror-RoBERTa-base, Encoding=bi-encoder2021.09 | 77.08 | — | 82.34 | — | 79.57 | — | — | — | — | 79.66 | — | — | |
| ALBEF14MBase Model=ALBEF14M, Modification=Fine-tune (None)2022.12 | 76.5 | 78.9 | 79.4 | 73.4 | 78.2 | — | — | — | 77.5 | 77.31 | — | — | |
| TENCBase Model=DeCLUTR-RoBERTa-base, Encoding=cross-encoder2021.09 | 75.55 | — | 84.02 | — | 83.41 | — | — | — | — | 80.99 | — | — | |
| Mirror-RoBERTa-baseBase Model=Mirror-RoBERTa-base2021.09 | 75.44 | — | 78.89 | — | 73.73 | — | — | — | — | 76.02 | — | — | |
| TENCBase Model=DeCLUTR-RoBERTa-base, Encoding=bi-encoder2021.09 | 74.55 | — | 83.64 | — | 82.87 | — | — | — | — | 80.35 | — | — | |
| CLIPBase Model=CLIP, Modification=Fine-tune (None)2022.12 | 74.3 | 73.3 | 72.8 | 68.4 | 74.5 | — | — | — | 73.8 | 72.85 | — | — | |
| Contrastive-Tension-BERT-baseBase Model=Contrastive-Tension-BERT-base2021.09 | 72.86 | — | 78.95 | — | 69.73 | — | — | — | — | 73.85 | — | — | |
| DeCLUTR-RoBERTa-baseBase Model=DeCLUTR-RoBERTa-base2021.09 | 72.86 | — | 78.68 | — | 74.77 | — | — | — | — | 75.44 | — | — | |
| TENCBase Model=Contrastive-Tension-BERT-base, Encoding=bi-encoder2021.09 | 72.54 | — | 80.96 | — | 68.31 | — | — | — | — | 73.94 | — | — | |
| BIBERTBitwidth=1-bit2024.05 | 72.5 | 88.7 | 84.8 | 66.1 | 72.6 | — | 57.4 | 25.4 | 33.6 | 63.2 | — | — | |
| TENCBase Model=Contrastive-Tension-BERT-base, Encoding=cross-encoder2021.09 | 71.33 | — | 81.22 | — | 74.17 | — | — | — | — | 75.57 | — | — | |
| Linear FTnormTrainable Parameters=0.03%, Backbone=RoBERTa-Large2023.05 | 71.1 | 95.1 | 87.8 | 88.4 | 92.5 | — | 53.4 | 47 | 75.3 | 76.3 | — | — | |
| BINARYBERTBitwidth=1-bit2024.05 | 68.3 | 53.2 | 66.2 | 35.6 | 51.5 | — | 52.7 | 0 | 6.1 | 41 | — | — | |
| ChatGPT2025.07 | 66 | 92 | 78 | 89.3 | 84 | — | — | — | — | — | — | — | |
| Linear FTTrainable Parameters=0%, Backbone=RoBERTa-Large2023.05 | 54.5 | 83.7 | 75.6 | 52.4 | 67.4 | — | 55.1 | 31.2 | 69.9 | — | — | — | |
| ALBERT_Large + MLM/SOPBackbone=ALBERT_Large, Objective=MLM/SOP, FLOPS=x 19372023.09 | — | — | — | — | — | — | — | — | — | 90 | — | — | |
| BERT_Base + MLMBackbone=BERT_Base, Objective=MLM, FLOPS=x 12023.09 | — | — | — | — | — | — | — | — | — | 79.7 | — | — | |
| BERT_Base + RTSBackbone=BERT_Base, Objective=RTS, FLOPS=x 0.812023.09 | — | — | — | — | — | — | — | — | — | 79.9 | — | — | |
| BERT_Base + SLMBackbone=BERT_Base, Objective=SLM, FLOPS=x 12023.09 | — | — | — | — | — | — | — | — | — | 80.4 | — | — | |
| BERT_Large + MLM/NSPBackbone=BERT_Large, Objective=MLM/NSP, FLOPS=x 122023.09 | — | — | — | — | — | — | — | — | — | 83.3 | — | — | |
| BERT_Small + MLMBackbone=BERT_Small, Objective=MLM, FLOPS=x 0.122023.09 | — | — | — | — | — | — | — | — | — | 74.1 | — | — | |
| BERT_Small + RTSBackbone=BERT_Small, Objective=RTS, FLOPS=x 0.102023.09 | — | — | — | — | — | — | — | — | — | 75.4 | — | — | |
| BERT_Small + SLMBackbone=BERT_Small, Objective=SLM, FLOPS=x 0.122023.09 | — | — | — | — | — | — | — | — | — | 75.7 | — | — | |
| ELECTRA_Base + MLM/TDBackbone=ELECTRA_Base, Objective=MLM/TD, FLOPS=x 212023.09 | — | — | — | — | — | — | — | — | — | 85.7 | — | — | |
| ELECTRA_Large + MLM/TDBackbone=ELECTRA_Large, Objective=MLM/TD, FLOPS=x 1942023.09 | — | — | — | — | — | — | — | — | — | 88.6 | — | — |