Natural Language Understanding on GLUE (test val)
94MRPC AccuracyFull-FT
Evaluation results
| Method | Links | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Full-FTBackbone=RoBERTa-Large, %Param=100%2025.12 | 94 | 96.4 | 68 | 86.6 | 92.4 | 90.2 | 94.7 | 92.2 | 89.3 | |
| LoRABackbone=RoBERTa-Large, %Param=0.3%2025.12 | 93.4 | 96.2 | 68.2 | 87.4 | 92.6 | 90.6 | 94.9 | 91.6 | 89.4 | |
| GRASPBackbone=RoBERTa-Large, %Param=0.005%, K=642025.12 | 93 | 96.1 | 67 | 85.9 | 91.8 | 88.9 | 94.5 | 89.8 | 88.4 | |
| ModernALBERT-largeParams=120M, Seq.=10242025.12 | 92.7 | 95.5 | 66.4 | 88.44 | 92.1 | 88.9 | 93.7 | 92 | 88.72 | |
| Full-FTBackbone=RoBERTa-Base, %Param=100%2025.12 | 92.5 | 94.2 | 61.1 | 77.4 | 90.6 | 86.7 | 92.3 | 91.9 | 85.8 | |
| ResLoRAisbackbone=RoBERTa-large2024.02 | 92.39 | 95.64 | 65.54 | 83.03 | 91.97 | 89.97 | 94.34 | 90.91 | — | |
| ModernBERTParams=149M, Seq.=81922025.12 | 92.2 | 96 | 65.1 | 87.4 | 91.8 | 89.1 | 93.9 | 92.1 | 88.45 | |
| GRASPBackbone=RoBERTa-Base, %Param=0.015%, K=1282025.12 | 92.2 | 94.3 | 64.2 | 79.1 | 90.9 | 85.2 | 92 | 89.5 | 85.9 | |
| GTE-en-MLMParams=137M, Seq.=81922025.12 | 92.1 | 93.4 | 57 | 84.8 | 90.2 | 86.7 | 91.9 | 88.8 | 85.99 | |
| BitFitBackbone=RoBERTa-Base, %Param=0.09%2025.12 | 92 | 93.7 | 61.8 | 77.8 | 90.8 | 85 | 91.3 | 87.3 | 84.9 | |
| LoRArank=16, backbone=RoBERTa-large2024.02 | 91.96 | 95.99 | 67.49 | 83.03 | 92.14 | 90.59 | 94.2 | 91.68 | — | |
| ModernALBERT-baseParams=75M, Seq.=10242025.12 | 91.7 | 95.2 | 64.6 | 85.6 | 91.8 | 88.2 | 93 | 91.5 | 87.7 | |
| QuanTAParameters (%)=0.62%, Base Model=RoBERTa2024.05 | 91.67 | 93.81 | 62.08 | 77.26 | 90.68 | — | — | — | — | |
| ResLoRAmsbackbone=RoBERTa-large2024.02 | 91.64 | 95.76 | 65.8 | 81.95 | 91.11 | 88.74 | 94.34 | 87.14 | — | |
| GRASPBackbone=RoBERTa-Base, %Param=0.004%, K=322025.12 | 91.5 | 93.8 | 61.9 | 77.2 | 90.3 | 83.4 | 90.4 | 88.2 | 84.6 | |
| LoRAParameters (%)=0.71%, Base Model=RoBERTa2024.05 | 91.48 | 94.01 | 62.08 | 74.51 | 90.48 | — | — | — | — | |
| ResLoRAbsbackbone=RoBERTa-large2024.02 | 91.31 | 96.22 | 65.44 | 82.31 | 91.72 | 90.4 | 94.48 | 91.27 | — | |
| ADAMBYTES/STEP=494M, Pre-trained model=RoBERTa-Base, Evaluation protocol=Fine-tuning2026.02 | 91.3 | 94.57 | 62.24 | 79.42 | 90.92 | 87.18 | 92.33 | 92.28 | 86.28 | |
| LoRArank=4, backbone=RoBERTa-large2024.02 | 91.13 | 95.3 | 65.29 | 79.06 | 91.67 | 90.26 | 94.23 | 91.01 | — | |
| ModernALBERT-mediumParams=55M, Seq.=10242025.12 | 91 | 94 | 62.3 | 81.6 | 91.2 | 86.6 | 92 | 91 | 86.21 | |
| MELORA#Params=295k2024.02 | 90.93 | 95.41 | 64.09 | 86.64 | 91.93 | 87.2 | 93.17 | 90.77 | 87.52 | |
| RoBERTa_LARGERelease Year=20192024.09 | 90.9 | 96.4 | 68 | 86.6 | — | 90.2 | 94.7 | 92.2 | 88.43 | |
| VeRABackbone=RoBERTa-Large, %Param=0.02%2025.12 | 90.9 | 96.1 | 68 | 85.9 | 91.7 | — | 94.4 | — | — | |
| MELORA#Params=37k2024.02 | 90.69 | 94.95 | 64.07 | 86.28 | 91.08 | 86.21 | 92.7 | 89.26 | 86.91 | |
| TSRBYTES/STEP=20M, Pre-trained model=RoBERTa-Base, Evaluation protocol=Fine-tuning2026.02 | 90.38 | 93.92 | 61.32 | 77.98 | 90.72 | 86.88 | 92.95 | 90.16 | 85.54 | |
| REDBackbone=RoBERTa-Large, %Param=0.02%2025.12 | 90.3 | 96 | 68.1 | 86.2 | 91.3 | 89.5 | 93.5 | 88.9 | 87.9 | |
| GALOREBYTES/STEP=158M, Pre-trained model=RoBERTa-Base, Evaluation protocol=Fine-tuning2026.02 | 90.27 | 93.92 | 59.33 | 77.98 | 90.82 | 87.11 | 92.42 | 89.54 | 85.17 | |
| RoBERTaParams=125M, Seq.=5122025.12 | 90.2 | 94.8 | 63.6 | 78.7 | 91.2 | 87.6 | 92.8 | 91.9 | 86.35 | |
| ModernALBERT-tinyParams=50M, Seq.=10242025.12 | 90.2 | 93 | 58.4 | 81.2 | 90.4 | 84.6 | 91.3 | 90.5 | 84.95 | |
| AdaLora#Params=295k2024.02 | 90.19 | 94.49 | 61.64 | 85.19 | 91.16 | 87.34 | 93.08 | 90.14 | 86.65 | |
| DeltaLoRA#Params=295k2024.02 | 90.19 | 95.06 | 63.82 | 87 | 91.57 | 87.5 | 93.09 | 90.87 | 87.38 | |
| LoRA#Params=295k2024.02 | 89.92 | 94.38 | 62.43 | 85.92 | 91.36 | 86.91 | 92.64 | 90.78 | 86.79 | |
| LoRABackbone=RoBERTa-Base, %Param=0.3%2025.12 | 89.7 | 95.1 | 63.4 | 86.6 | 91.5 | 87.5 | 93.3 | 90.8 | 87.2 | |
| BERTParams=110M, Seq.=5122025.12 | 89.5 | 93.1 | 59 | 78.2 | 89.4 | 85.4 | 91.6 | 91.4 | 84.84 | |
| DeBERTav3Params=183M, Seq.=5122025.12 | 89.5 | 95.6 | 69.2 | 83.8 | 91.6 | 90 | 94 | 92.4 | 88.26 | |
| DyLoRA#Params=295k2024.02 | 89.46 | 94.26 | 61.12 | 84.47 | 91.06 | 86.33 | 92.22 | 90.17 | 86.14 | |
| REDBackbone=RoBERTa-Base, %Param=0.02%2025.12 | 89.2 | 93.9 | 61 | 78 | 90.4 | 83.9 | 90.7 | 87.2 | 84.3 | |
| MosaicBERT-128Params=137M, Seq.=1282025.12 | 89 | 93.5 | 58.2 | 83 | 90.3 | 85.6 | 91.4 | 92 | 85.38 | |
| LoHAbackbone=RoBERTa-large2024.02 | 88.85 | 95.76 | 60.57 | 55.23 | 90.21 | 89.63 | 94.03 | 89.93 | — | |
| AdaptersBackbone=RoBERTa-Base, %Param=0.3%2025.12 | 88.5 | 94.2 | 60.8 | 71.5 | 89.7 | 87.1 | 93.1 | 90.2 | 84.4 | |
| FT#Params=125.0M2024.02 | 88.23 | 94.26 | 64.57 | 84.11 | 90.56 | 87.51 | 92.73 | 91.96 | 86.74 | |
| BERT_LARGERelease Year=20192024.09 | 88 | 93.2 | 60.6 | 70.4 | — | 86.6 | 92.3 | 91.3 | 83.2 | |
| NomicBERT-2048Params=137M, Seq.=20482025.12 | 88 | 93 | 50 | 82 | 90 | 86 | 92 | 92 | 84.13 | |
| Expert (upper bound)Sparsity (k)=20%, Backbone=BERT-base2025.12 | 86.8 | — | 43.4 | 58.1 | — | — | 76.2 | — | 66.1 | |
| (IA)3Backbone=RoBERTa-Base, %Param=0.05%2025.12 | 86.4 | 93.4 | 57.8 | 73.5 | 88.5 | 85.4 | 91.1 | 88.5 | 83.1 | |
| LoKrbackbone=RoBERTa-large2024.02 | 81.41 | 94.5 | 55.22 | 52.71 | 83.35 | 87.08 | 92.44 | 88.3 | — | |
| AdaLoRAbackbone=RoBERTa-large2024.02 | 81.22 | 95.3 | 57.01 | 52.71 | 89.76 | 89.11 | 94.29 | 89.77 | — | |
| Fisher-100Sparsity (k)=20%, Backbone=BERT-base, Fisher Samples=1002025.12 | 80.1 | — | 22.9 | 47.7 | — | — | 42.6 | — | 48.3 | |
| Fisher-500Sparsity (k)=20%, Backbone=BERT-base, Fisher Samples=5002025.12 | 80 | — | 23.2 | 46.9 | — | — | 40.8 | — | 47.7 | |
| UMTAMSparsity (k)=20%, Backbone=BERT-base2025.12 | 79.4 | — | 17.7 | 45.1 | — | — | 45.4 | — | 46.9 | |
| TIES (baseline)Sparsity (k)=20%, Backbone=BERT-base2025.12 | 79.2 | — | 12.3 | 46.2 | — | — | 46 | — | 45.9 | |
| OTA-styleSparsity (k)=20%, Backbone=BERT-base, Curvature Proxy=Adam's second moment estimate (vT)2025.12 | 79 | — | 25.7 | 46.6 | — | — | 41.4 | — | 48.2 | |
| Fisher-TrainSparsity (k)=20%, Backbone=BERT-base, Fisher Samples=10002025.12 | 78.5 | — | 12.7 | 45.1 | — | — | 46.4 | — | 45.7 | |
| GPT-4o mini + SENSEBackbone=GPT-4o mini, SENSE=true2024.09 | 76.47 | 92.08 | 67.22 | 88.45 | — | 78.2 | 93.3 | 73 | 81.25 | |
| GPT-3.5 turbo + SENSEBackbone=GPT-3.5 turbo, SENSE=true2024.09 | 75.49 | 92.2 | 64.57 | 84.12 | — | 64.6 | 83.2 | 77.2 | 77.34 | |
| LLaMA3-70B + SENSEBackbone=LLaMA3-70B, SENSE=true2024.09 | 74.04 | 95.18 | 65.53 | 85.56 | — | 73.1 | 92.8 | 76.5 | 80.25 | |
| LLaMA3-70BSENSE=false2024.09 | 73.52 | 95.64 | 63.9 | 84.48 | — | 71.9 | 91.3 | 74.6 | 79.34 | |
| GPT-3.5 turboSENSE=false2024.09 | 73.28 | 91.86 | 63.5 | 81.81 | — | 61.8 | 82.4 | 73.4 | 75.44 | |
| GPT-4o miniSENSE=false2024.09 | 72.3 | 91.63 | 65.49 | 87.36 | — | 73.9 | 92.3 | 73 | 79.43 |