Natural Language Understanding on GLUE
156SST-2Fast Post-Training Pruning Framework
Evaluation results
| Method | Links | |||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Fast Post-Training Pruning FrameworkBatch size=256, Backbone=BERT_BASE, Hardware=NVIDIA V100 GPU, Framework=PyTorch, Accuracy degradation constraint=at most 1%2022.03 | 156 | 134 | 155 | 153 | 154 | 154 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Fast Post-Training Pruning FrameworkBatch size=32, Backbone=BERT_BASE, Hardware=NVIDIA V100 GPU, Framework=PyTorch, Accuracy degradation constraint=at most 1%2022.03 | 123 | 127 | 136 | 142 | 142 | 134 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Fine-TuningBackbone=DeBERTa-XXL, Trainable Parameters=1500.0M, Source=Published in prior works2021.06 | 97.2 | 91.8 | 92 | 96 | 92.7 | 92.9 | 91.1 | — | 72 | 93.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ALBERTSource=Leaderboard (September, 2019)2019.09 | 97.1 | — | 93.4 | 99.2 | 74.2 | 92.5 | 89.4 | — | 69.1 | 89.2 | — | — | — | 91.3 | 91 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ModernBERT-largeModel size category=Large Models, Parameters range=305-395M parameters2025.07 | 97.1 | 90.8 | 91.7 | 95.2 | 92.7 | 92.8 | 90.4 | — | 71.4 | 92.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DeBERTa-v2-XLModel size category=XL Models, Parameters range=750-1565M parameters2025.07 | 97.1 | 91.7 | 91.7 | 95.9 | 92.6 | 92.5 | 90.7 | — | 75.3 | 89.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Ettin-Enc-1bModel size category=XL Models, Parameters range=750-1565M parameters2025.07 | 97.1 | 91.8 | 94.4 | 96 | 93 | 93.2 | 91.6 | — | 74.4 | 93.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| XLNetPre-training modality=Natural Language2022.02 | 97 | 90.8 | 90.8 | 94.9 | 92.3 | — | — | — | — | 85.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LoRABackbone=DeBERTa-XXL, Trainable Parameters=4.7M2021.06 | 96.9 | 91.9 | 92.6 | 96 | 92.9 | 93 | 91.3 | — | 72.4 | 94.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ELECTRAPre-training modality=Natural Language2022.02 | 96.9 | 90.9 | 90.8 | 95 | 92.4 | — | — | — | — | 88 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| XLNETSource=Leaderboard (September, 2019)2019.09 | 96.8 | — | 93 | 98.6 | 90.3 | 91.6 | 88.4 | — | 67.8 | 86.3 | — | — | — | 90.2 | 89.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| UNIMOPre-training modality=Multimodal2022.02 | 96.8 | 89.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DeBERTaPre-training modality=Natural Language2022.02 | 96.8 | 91.1 | 91.9 | 95.3 | 92.3 | — | — | — | — | 88.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| RoBERTaSource=Leaderboard (September, 2019)2019.09 | 96.7 | — | 92.1 | 98.9 | 90.2 | 92.2 | 88.5 | — | 67.8 | 88.2 | — | — | — | 90.8 | 90.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FLOATERModel Scale=Large model2020.03 | 96.7 | 90.4 | 91.4 | 94.8 | 92.2 | 92.5 | — | — | 69 | 87 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Ettin-Enc-400mModel size category=Large Models, Parameters range=305-395M parameters2025.07 | 96.7 | 91.3 | 93.6 | 95.2 | 93 | 92.7 | 90.8 | — | 71.3 | 92.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| RoCoFT3-RowLM=RoBERTa-large, #TTPS=0.666M2024.10 | 96.69 | 90.98 | 91.05 | 94.85 | 90.82 | — | — | — | — | 87.83 | — | — | — | — | — | — | — | — | — | — | — | — | — | 67.39 | — | 92.19 | 86.11 | — | — | — | — | — | 92.1 | — | 92.1 | — | — | — | — | |
| LoRALM=RoBERTa-large, #TTPS=1.84M2024.10 | 96.67 | 90.76 | 87.5 | 95 | 90.15 | — | — | — | — | 79.78 | — | — | — | — | — | — | — | — | — | — | — | — | — | 64.47 | — | 88.19 | 86.91 | — | — | — | — | — | 91.44 | — | 91.66 | — | — | — | — | |
| RoCoFT3-ColumnLM=RoBERTa-large, #TTPS=0.666M2024.10 | 96.67 | 91.13 | 89.88 | 94.85 | 91.38 | — | — | — | — | 87.82 | — | — | — | — | — | — | — | — | — | — | — | — | — | 67.18 | — | 91.47 | 87.12 | — | — | — | — | — | 92.31 | — | 92.52 | — | — | — | — | |
| RoCoFT1-RowLM=RoBERTa-large, #TTPS=0.222M2024.10 | 96.63 | 90.73 | 89.97 | 94.2 | 90.17 | — | — | — | — | 85.31 | — | — | — | — | — | — | — | — | — | — | — | — | — | 65.7 | — | 90.79 | 86.15 | — | — | — | — | — | 92.07 | — | 91.81 | — | — | — | — | |
| BARTModel Size=Large2019.10 | 96.6 | — | 90.4 | 94.9 | 92.5 | 91.2 | — | — | 62.8 | 87 | — | — | — | 89.9 | 90.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AdapterLBackbone=RoBERTa-large, Trainable Parameters=0.8M, Experimental Setup=Similar to Houlsby et al. (2019)2021.06 | 96.6 | 90.5 | 89.7 | 94.8 | 91.7 | 91.9 | 87.9 | — | 67.8 | 80.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| OFAFinetuning protocol=RTE and MRPC finetuned from MNLI checkpoint2022.02 | 96.6 | 90.2 | 91.7 | 94.8 | 92.5 | — | — | — | — | 91 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| 32-bit ReplicationPrecision=32-bit2022.08 | 96.6 | 90.3 | 90.4 | 94.8 | 92.3 | 92 | 88.83 | — | 68.8 | 85.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| RoCoFT1-ColumnLM=RoBERTa-large, #TTPS=0.222M2024.10 | 96.6 | 90.81 | 89.12 | 94.17 | 90.17 | — | — | — | — | 85.71 | — | — | — | — | — | — | — | — | — | — | — | — | — | 64.89 | — | 90.24 | 85.83 | — | — | — | — | — | 92.1 | — | 91.96 | — | — | — | — | |
| AdapterLBackbone=RoBERTa Large, #Paras=1.8M2025.12 | 96.6 | — | 89.7 | 94.8 | 91.5 | — | — | — | — | 80.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | 67.8 | 65.8 | — | — | — | — | — | — | — | — | — | 91.9 | — | — | — | — | |
| MDPD (SHERL-based, T5-large)Params. (%)=0.81, Memory (G) Train=8.0, Memory (G) Test=2.80, Time (ms)=415.3, backbone=T5-large2026.04 | 96.5 | 88.9 | 93.5 | 94.7 | 90.2 | 92.4 | 88 | — | 66.2 | 81.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| RoBERTaModel Size=Large2019.10 | 96.4 | — | 90.9 | 94.7 | 92.2 | 92.4 | — | — | 68 | 86.6 | — | — | — | 90.2 | 90.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ROBERTaModel Scale=Large model2020.03 | 96.4 | 90.2 | 90.9 | 94.7 | 92.2 | 92.4 | — | — | 68 | 86.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Fine-TuningBackbone=RoBERTa-large, Trainable Parameters=355.0M, Source=Published in prior works2021.06 | 96.4 | 90.2 | 90.9 | 94.7 | 92.2 | 92.4 | 88.9 | — | 68 | 86.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| RoBERTaPre-training modality=Natural Language2022.02 | 96.4 | 90.2 | 90.9 | 93.9 | 92.2 | — | — | — | — | 86.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| RoBERTa-LBackbone=Large, Note=Uses 10x training text tokens than ours2022.06 | 96.4 | 90.2 | — | 94.7 | — | — | — | — | — | 86.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | 68 | — | 90.9 | 92.2 | — | — | — | — | — | — | — | — | — | — | — | — | |
| 32-bit BaselinePrecision=32-bit2022.08 | 96.4 | 90.4 | 90.1 | 94.9 | 92.2 | 93 | 88.61 | — | 67.4 | 84.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Vector-wiseFFN Precision=8-bit, Rest Precision=16-bit, Mixed-precision decomposition=false, Protocol=fine-tuning2022.08 | 96.4 | 90.2 | 91 | 94.7 | 92.3 | 91.9 | 88.81 | — | 68.6 | 85.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| RoBERTa-large (Original)Backbone=RoBERTa-large2022.12 | 96.4 | 90.2 | 90.9 | 94.7 | 92.2 | 92.4 | 88.9 | — | 68 | 86.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| TencentPretrain (RoBERTa-large)Backbone=RoBERTa-large, Pre-train from scratch=true, Fine-tuning=true2022.12 | 96.4 | 90.4 | 90.2 | 94.7 | 92.1 | 92.5 | 88.7 | — | 67 | 86.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FactorPrecision=4-bit2023.09 | 96.4 | 90.1 | 92.7 | 94.7 | 92.2 | 92.3 | — | — | 68.1 | 85.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Full-FT#Trainable Params=355.0M, #Foundation Model Params=100%, Training Speed up=0.7x2024.06 | 96.4 | 90.4 | — | 94.7 | 92.2 | — | 93.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Full Parameter Fine-tuningBackbone=RoBERTa Large, #Paras=356M2025.12 | 96.4 | — | 90.9 | 95 | 92 | — | — | — | — | 86.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | 68 | 66.5 | — | — | — | — | — | — | — | — | — | 92.4 | — | — | — | — | |
| AdaptersLM=RoBERTa-large, #TTPS=19.77M2024.10 | 96.37 | 91 | 89.88 | 94.31 | 91.19 | — | — | — | — | 85.25 | — | — | — | — | — | — | — | — | — | — | — | — | — | 65.33 | — | 90.23 | 87.11 | — | — | — | — | — | 92.42 | — | 92.58 | — | — | — | — | |
| AdapterLBackbone=RoBERTa-large, Trainable Parameters=0.8M, Experimental Setup=Similar to Houlsby et al. (2019), Identity Variant=Repeat setup2021.06 | 96.3 | 90.3 | 87.7 | 94.7 | 91.5 | 91.5 | 86.4 | — | 66.3 | 72.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| RoBERTa^ASABackbone architecture=RoBERTa, ASA=true2022.06 | 96.3 | 88 | — | 93.6 | 73.7 | 89.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AdapterBackbone=RoBERTa_LARGE, # Trainable Parameters=0.8M, # Total Parameters=100%2021.10 | 96.3 | — | — | 94.7 | — | — | — | — | 66.3 | 72.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AdamWPrecision=32-bit2023.09 | 96.3 | 90.2 | 93.2 | 94.9 | 92.2 | 92.3 | — | — | 66.9 | 85.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AdafactorPrecision=32-bit2023.09 | 96.3 | 90.4 | 92.8 | 94.7 | 92.2 | 92.3 | — | — | 67.3 | 85.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AdafactorPrecision=32-bit, quantize optimizer states for embedding layers=false2023.09 | 96.3 | 90.5 | 92.9 | 94.8 | 92.2 | 92.2 | — | — | 68.2 | 87 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AdamWPrecision=4-bit2023.09 | 96.3 | 90.2 | 92.8 | 94.5 | 92 | 92.5 | — | — | 67.3 | 85.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Adapter#Trainable Params=0.8M, #Foundation Model Params=100%, Training Speed up=1x2024.06 | 96.3 | 90.8 | — | 94.7 | 91.5 | — | 93.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| CompacterBackbone=RoBERTa Large, #Paras=0.9M2025.12 | 96.3 | — | 87.7 | 94.7 | 91 | — | — | — | — | 88.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | 66.3 | 65 | — | — | — | — | — | — | — | — | — | 91.5 | — | — | — | — | |
| MDPD (UniPT-based, T5-large)Params. (%)=1.02, Memory (G) Train=10.2, Memory (G) Test=2.82, Time (ms)=418.7, backbone=T5-large2026.04 | 96.3 | 88.4 | 92.8 | 94.5 | 90 | 92.3 | 87.6 | — | 66.3 | 79.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PROPETL AdapterLM=RoBERTa-large, #TTPS=5.40M2024.10 | 96.27 | 91.37 | 89.71 | 94.8 | 90.67 | — | — | — | — | 87.69 | — | — | — | — | — | — | — | — | — | — | — | — | — | 65.55 | — | 91.15 | 87.74 | — | — | — | — | — | 91.67 | — | 91.92 | — | — | — | — | |
| PiSSAParams=1.33M2024.04 | 96.22 | 90.37 | 91.5 | 94.43 | 92.33 | 92 | 89.83 | — | 73.12 | 88.69 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DeBERTa-v3-large# of Parameters=300M, GPU Memory=600MB2025.04 | 96.21 | 91.64 | 92.85 | 95.51 | 92 | 92.93 | — | — | 75.72 | 90.61 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LoRABackbone=RoBERTa-large, Trainable Parameters=0.8M2021.06 | 96.2 | 90.6 | 90.9 | 94.9 | 91.6 | 92.6 | 89 | — | 68.2 | 87.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AdapterHBackbone=RoBERTa-large, Trainable Parameters=6.0M, Experimental Setup=Similar to Houlsby et al. (2019)2021.06 | 96.2 | 89.9 | 88.7 | 94.7 | 92.1 | 91 | 87.8 | — | 66.5 | 83.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LoRABackbone=RoBERTa-large, Trainable Parameters=0.8M, Experimental Setup=Similar to Houlsby et al. (2019)2021.06 | 96.2 | 90.6 | 90.2 | 94.8 | 91.6 | 92.3 | 88.6 | — | 68.2 | 85.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PSQFFN Precision=8-bit, Rest Precision=16-bit, Mixed-precision decomposition=false, Protocol=fine-tuning2022.08 | 96.2 | 89.9 | 90.4 | 94.5 | 92 | 91.9 | 88.65 | — | 67.5 | 86.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LoRATotal Parameters=100%2021.10 | 96.2 | — | — | 94.8 | — | — | — | — | — | 85.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LoRABackbone=RoBERTa_LARGE, # Trainable Parameters=0.8M, # Total Parameters=100%2021.10 | 96.2 | — | — | 94.8 | — | — | — | — | 68.2 | 85.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AdamWPrecision=8-bit, quantize optimizer states for embedding layers=false2023.09 | 96.2 | 90.4 | 93.2 | 94.8 | 92.2 | 92.2 | — | — | 68 | 84.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LoRA#Trainable Params=0.8M, #Foundation Model Params=100%, Training Speed up=1x2024.06 | 96.2 | 90.6 | — | 94.9 | 91.6 | — | 93.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AdapterPBackbone=RoBERTa Large, #Paras=0.9M2025.12 | 96.2 | — | 88.7 | 94.7 | 91.2 | — | — | — | — | 83.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | 66.5 | 65.3 | — | — | — | — | — | — | — | — | — | 91 | — | — | — | — | |
| Parallel AdapterBackbone=RoBERTa Large, #Paras=4.8M2025.12 | 96.2 | — | 90.2 | 94.8 | 91.8 | — | — | — | — | 85.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | 68.2 | 66 | — | — | — | — | — | — | — | — | — | 92.3 | — | — | — | — | |
| PROPETL PrefixLM=RoBERTa-large, #TTPS=26.85M2024.10 | 96.17 | 90.33 | 90.04 | 94.73 | 89.3 | — | — | — | — | 79.71 | — | — | — | — | — | — | — | — | — | — | — | — | — | 62.24 | — | 91.92 | 86.3 | — | — | — | — | — | 90.49 | — | 90.7 | — | — | — | — | |
| MoSLoRALM=RoBERTa-large, #TTPS=3.23M2024.10 | 96.17 | 90.29 | 89.96 | 94.73 | 90.12 | — | — | — | — | 82.41 | — | — | — | — | — | — | — | — | — | — | — | — | — | 67.27 | — | 92.67 | 87.68 | — | — | — | — | — | 91.72 | — | 90.97 | — | — | — | — | |
| SynFlowBackbone=OPT-1.3b, Trainable Parameters (%)=0.18%, #=12024.12 | 96.1 | — | 78.68 | 90.85 | 88.56 | 81.66 | 87.17 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AdapterHBackbone=RoBERTa-large, Trainable Parameters=3.0M, Experimental Setup=Similar to Houlsby et al. (2019)2021.06 | 96.1 | 90.2 | 90.2 | 94.8 | 91.9 | 92.1 | 88.4 | — | 68.3 | 83.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MAM AdapterTotal Parameters=100%2021.10 | 96.1 | — | — | 94.7 | — | — | — | — | — | 80.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DSEETotal Parameters=70%2021.10 | 96.1 | — | — | 94.4 | — | — | — | — | — | 84.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DSEEBackbone=RoBERTa_LARGE, # Trainable Parameters=0.8M, # Total Parameters=70%2021.10 | 96.1 | — | — | 94.4 | — | — | — | — | 67.2 | 84.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Adapter-FFNBackbone=RoBERTa-large, # Param=0.8M2024.02 | 96.1 | 90.3 | 90.5 | 94.3 | 91.3 | 90.2 | 87.7 | — | 64.4 | 84.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AdaLoRAParams=1.27M2024.04 | 96.1 | 90.76 | 90.69 | 94.55 | 92.23 | 91.84 | 89.46 | — | 71.45 | 88.09 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BitfitLM=RoBERTa-large, #TTPS=0.222M2024.10 | 96.1 | 89.98 | 90.93 | 94.47 | 89.48 | — | — | — | — | 87.73 | — | — | — | — | — | — | — | — | — | — | — | — | — | 67.01 | — | 92.13 | 86.43 | — | — | — | — | — | 93.38 | — | 91.93 | — | — | — | — | |
| FAABackbone=RoBERTa Base, #Paras=0.6M2025.12 | 96.1 | — | 91 | 93.7 | 94.5 | — | — | — | — | 80 | — | — | — | — | — | — | — | — | — | — | — | — | — | 63.3 | 67 | — | — | — | — | — | — | — | — | — | 91.4 | — | — | — | — | |
| AdapterHBackbone=RoBERTa Large, #Paras=1.8M2025.12 | 96.1 | — | 90.2 | 94.8 | 91.8 | — | — | — | — | 83.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | 68.3 | 65.5 | — | — | — | — | — | — | — | — | — | 92.1 | — | — | — | — | |
| SM3Precision=32-bit2023.09 | 96 | 90.6 | 90.5 | 94.2 | 89.5 | 91.4 | — | — | 62.3 | 85.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FTBackbone=RoBERTa-large, # Param=355M2024.02 | 96 | 88.8 | 91.7 | 93.8 | 91.5 | 92.6 | 88.5 | — | 68.2 | 85.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LoRABackbone=RoBERTa-large, # Param=0.8M2024.02 | 96 | 90.2 | 89.8 | 94.7 | 90.7 | 91.7 | 88.1 | — | 65.5 | 86.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| REDBackbone=RoBERTa-large, # Param=0.05M2024.02 | 96 | 89.5 | 90.3 | 93.5 | 88.8 | 91.3 | 87.9 | — | 68.1 | 86.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LoRABackbone=RoBERTa Large, #Paras=0.8M2025.12 | 96 | — | 91.5 | 94.4 | 91.5 | — | — | — | — | 87.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | 67.1 | 66.2 | — | — | — | — | — | — | — | — | — | 91.9 | — | — | — | — | |
| FAABackbone=RoBERTa Large, #Paras=1.8M2025.12 | 96 | — | 90 | 94.7 | 92.8 | — | — | — | — | 86.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | 69 | 67.5 | — | — | — | — | — | — | — | — | — | 91.8 | — | — | — | — | |
| ModernBERT-baseModel size category=Base Models, Parameters range=86-150M parameters2025.07 | 96 | 89.1 | 92.2 | 93.9 | 92.1 | 91.8 | 88.4 | — | 65.1 | 87.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LoRABackbone=OPT-1.3b, Trainable Parameters (%)=0.18%, #=12024.12 | 95.99 | — | 83.33 | 92.48 | 89.97 | 89.03 | 90.16 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AdafactorBackbone=OPT-1.3B2025.12 | 95.98 | 87.02 | 86.75 | 92.34 | 87.42 | — | — | — | 64.06 | 74.12 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PROPETL LORALM=RoBERTa-large, #TTPS=4.19M2024.10 | 95.93 | 90.53 | 87.31 | 94.93 | 90.93 | — | — | — | — | 83.57 | — | — | — | — | — | — | — | — | — | — | — | — | — | 61.9 | — | 89.87 | 88.05 | — | — | — | — | — | 91.38 | — | 91.66 | — | — | — | — | |
| Light-PEFT (LoRA)#Trainable Params=0.3M, #Foundation Model Params=72%, Training Speed up=1.4x2024.06 | 95.9 | 89.4 | — | 93.6 | 89.7 | — | 92.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MAM AdapterLM=RoBERTa-large, #TTPS=4.20M2024.10 | 95.81 | 90.62 | 90.12 | 94.31 | 90.87 | — | — | — | — | 86.62 | — | — | — | — | — | — | — | — | — | — | — | — | — | 67.39 | — | 92.07 | 86.65 | — | — | — | — | — | 92.18 | — | 92.44 | — | — | — | — | |
| Light-PEFT (LoRA)#Trainable Params=0.3M, #Foundation Model Params=67%, Training Speed up=1.6x2024.06 | 95.8 | 89 | — | 93.5 | 89.2 | — | 91.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Ettin-Enc-150mModel size category=Base Models, Parameters range=86-150M parameters2025.07 | 95.8 | 89.2 | 92.6 | 94 | 92.4 | 92.2 | 88.9 | — | 66.9 | 87.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| UniPT (T5-large)Params. (%)=0.92, Memory (G) Train=9.1, Memory (G) Test=2.82, Time (ms)=795.1, backbone=T5-large2026.04 | 95.8 | 88.2 | 92 | 94.2 | 89.7 | 92 | 87.2 | — | 65.7 | 79.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SHERL (T5-large)Params. (%)=0.64, Memory (G) Train=7.1, Memory (G) Test=2.80, Time (ms)=667.4, backbone=T5-large2026.04 | 95.8 | 88.6 | 92.9 | 94.2 | 89.6 | 92.1 | 87.5 | — | 65.6 | 80.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DoRAParams=1.27M2024.04 | 95.79 | 90.29 | 90.93 | 94.1 | 92.07 | 91.79 | 88.98 | — | 70.85 | 86.04 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Prefix-tuningLM=RoBERTa-large, #TTPS=2.03M2024.10 | 95.76 | 89.3 | 88.24 | 93.32 | 88.88 | — | — | — | — | 74.01 | — | — | — | — | — | — | — | — | — | — | — | — | — | 59.01 | — | 89.37 | 85.45 | — | — | — | — | — | 91.07 | — | 90.92 | — | — | — | — | |
| LST (T5-large)Params. (%)=1.23, Memory (G) Train=12.2, Memory (G) Test=2.88, Time (ms)=832.9, backbone=T5-large2026.04 | 95.7 | 88.6 | 91.6 | 94.1 | 89.7 | 92.4 | 87.1 | — | 65.3 | 79.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AladaBackbone=OPT-1.3B2025.12 | 95.68 | 87.99 | 89.09 | 92.38 | 88.69 | — | — | — | 64.56 | 78.82 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LoRAParams=1.33M, initialization=Kaiming2024.04 | 95.64 | 89.96 | 90.28 | 93.84 | 92.03 | 91.68 | 88.62 | — | 70.69 | 84.84 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Full FTParams=184M2024.04 | 95.63 | 89.9 | 89.46 | 94.03 | 92.4 | 91.6 | 88.25 | — | 69.19 | 83.75 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PAdapterParams=1.18M2024.04 | 95.61 | 90.33 | 89.46 | 94.29 | 92.04 | 91.54 | 88.41 | — | 68.77 | 85.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| XLNetModel Size=Large2019.10 | 95.6 | — | 89.2 | 93.9 | 91.8 | 91.8 | — | — | 63.6 | 83.8 | — | — | — | 89.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MT-DNNSource=In literature (April, 2019)2019.09 | 95.6 | — | 90 | — | 72.4 | 88.3 | 82.2 | — | 61.5 | 75.5 | — | — | — | 86.7 | 86 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |