Natural Language Understanding on GLUE (CoLA, STS-B, MRPC, RTE, QQP, QNLI, SST-2, MNLI)
89.5Average Score (GLUE)PiSSA
Evaluation results
| Method | Links | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| PiSSAMemory=730.5 MB, Backbone=DeBERTaV3-Base2026.05 | 89.5 | — | — | 92.3 | 94.3 | 95.9 | 72.6 | 91.9 | 91.7 | 87 | — | 90.4 | — | — | |
| CERSAMemory=728.6 MB, Backbone=DeBERTaV3-Base2026.05 | 89.5 | — | — | 92.4 | 94.4 | 96 | 72.3 | 91.7 | 92 | 87 | — | 90.3 | — | — | |
| SVFTMemory=1108.6 MB, Backbone=DeBERTaV3-Base2026.05 | 89 | — | — | 91.5 | 94.3 | 95.4 | 72.6 | 91.8 | 89 | 87.2 | — | 90 | — | — | |
| SVFitMemory=1096.3 MB, Backbone=DeBERTaV3-Base2026.05 | 88.6 | — | — | 90.2 | 94.3 | 95.4 | 71 | 91.8 | 88.8 | 87.4 | — | 89.7 | — | — | |
| LoRAMemory=730.5 MB, Backbone=DeBERTaV3-Base2026.05 | 88.5 | — | — | 92 | 93.9 | 95 | 69.8 | 91.6 | 90 | 85.2 | — | 90.7 | — | — | |
| FTMemory=2814.0 MB, Backbone=DeBERTaV3-Base2026.05 | 88.3 | — | — | 92.4 | 94 | 95.6 | 69.2 | 91.6 | 89.5 | 83.8 | — | 89.9 | — | — | |
| GenFTBackbone=RoBERTaBase, Param. (M)=0.242025.05 | 85.87 | — | — | 90.59 | 92.49 | 94.84 | 63.38 | 90.69 | 89.95 | 77.62 | — | 87.38 | — | — | |
| RoBERTaLearning Strategy=reference2026.06 | 85.5 | — | — | 91.7 | 92.7 | 94.8 | 61.9 | 90.2 | 88.5 | 76.9 | — | 87.6 | — | — | |
| PiSSABackbone=RoBERTaBase, Param. (M)=0.302025.05 | 85.42 | — | — | 88.37 | 91.8 | 93.92 | 66.03 | 90.4 | 88.73 | 78.7 | — | 85.41 | — | — | |
| GaLoreBackbone=RoBERTaBase2025.05 | 85.23 | — | — | 91.06 | 92.24 | 94.04 | 60.35 | 90.73 | 87.01 | 79.42 | — | 87 | — | — | |
| PropulsionBackbone=RoBERTaBase, Param. (M)=0.092025.05 | 84.95 | — | — | 89.11 | 92.79 | 93.18 | 61.76 | 91.37 | 89.34 | 75.66 | — | 86.41 | — | — | |
| CorDABackbone=RoBERTaBase, Param. (M)=0.292025.05 | 84.83 | — | — | 91.1 | 91.49 | 93.12 | 59.6 | 90.17 | 89.71 | 76.17 | — | 87.28 | — | — | |
| LISABackbone=RoBERTaBase, Param. (M)=53.172025.05 | 84.55 | — | — | 88.37 | 90.72 | 94.04 | 62.29 | 89.72 | 89.22 | 77.26 | — | 84.79 | — | — | |
| FiRABackbone=RoBERTaBase2025.05 | 84.5 | — | — | 90.3 | 91.69 | 93.58 | 59.83 | 90.33 | 88.97 | 75.09 | — | 86.25 | — | — | |
| PROPETLv1Backbone=RoBERTaBase, Param. (M)=1.872025.05 | 84.46 | — | — | 89.22 | 92.56 | 92.18 | 63.11 | 91.33 | 85.25 | 75.54 | — | 86.49 | — | — | |
| AdapterSBackbone=RoBERTaBase, Param. (M)=7.412025.05 | 84.31 | — | — | 90.81 | 90.84 | 92.14 | 60.32 | 90.25 | 89.24 | 73.56 | — | 87.33 | — | — | |
| BitFitBackbone=RoBERTaBase, Param. (M)=0.0862025.05 | 84.2 | — | — | 88.12 | 91.09 | 92.67 | 61.38 | 90.34 | 88.22 | 75.58 | — | 84.64 | — | — | |
| LoRABackbone=RoBERTaBase, Param. (M)=0.892025.05 | 83.99 | — | — | 88.83 | 92.02 | 92.4 | 60.09 | 90.66 | 88.5 | 72.92 | — | 86.54 | — | — | |
| Full Precision#Bits=32/32, Size (MB)=418, Data Augmentation=false2026.04 | 83.9 | 84.9 | 85.5 | 91.4 | 88.6 | 93.2 | 59.7 | 89.8 | 86.2 | 72.2 | — | — | — | — | |
| Flexformer_nLearning Strategy=distilled2026.06 | 83.7 | — | — | 91.1 | 91 | 93.6 | 63.3 | 88.3 | 85.8 | 71.5 | — | 85.1 | — | — | |
| MAMAdapterBackbone=RoBERTaBase, Param. (M)=46.782025.05 | 83.67 | — | — | 88.31 | 90.19 | 93.19 | 58.42 | 90.74 | 89.31 | 72.62 | — | 86.63 | — | — | |
| Full FTBackbone=RoBERTaBase, Param. (M)=124.62025.05 | 83.56 | — | — | 90.18 | 91.17 | 92.89 | 59.84 | 90.48 | 85.24 | 72.43 | — | 86.27 | — | — | |
| AdaLoRABackbone=RoBERTaBase, Param. (M)=1.032025.05 | 83.45 | — | — | 88.58 | 91.43 | 91.69 | 59.82 | 90.83 | 88.99 | 70.04 | — | 86.26 | — | — | |
| PROPETLv3Backbone=RoBERTaBase, Param. (M)=1.772025.05 | 83.45 | — | — | 88.9 | 92.06 | 92.54 | 61.72 | 90.76 | 87.42 | 67.39 | — | 86.84 | — | — | |
| HedgehogLearning Strategy=distilled2026.06 | 83.3 | — | — | 91.2 | 91.2 | 93.3 | 62.5 | 87.7 | 86 | 71 | — | 83.7 | — | — | |
| ModernBERT-base2026.06 | 83.01 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| (IA)3Backbone=RoBERTaBase, Param. (M)=0.662025.05 | 82.85 | — | — | 87.99 | 90.88 | 92.02 | 59.58 | 90.3 | 87 | 71.12 | — | 83.95 | — | — | |
| BEBERT#Bits=1/4, Size (MB)=33.0, Data Augmentation=true2026.04 | 82.4 | 84.7 | 84.7 | 91.4 | 91.5 | 93.4 | 56.5 | — | 85.2 | 71.5 | — | — | — | — | |
| EvoGMModel=FLAN-T52026.05 | 82.4 | — | — | 84.8 | 88.1 | 90 | 71.1 | 80.9 | 78.9 | 82.2 | — | 82.8 | — | — | |
| Full Precision#Bits=32/32, Size (MB)=418, Data Augmentation=true2026.04 | 82.3 | 84.9 | 85.5 | 91.4 | 92.1 | 93.2 | 59.7 | 90.1 | 86.3 | 72.2 | — | — | — | — | |
| PROPETLv2Backbone=RoBERTaBase, Param. (M)=10.492025.05 | 82.26 | — | — | 88.54 | 91.51 | 91.36 | 60.18 | 90.3 | 86.73 | 63.31 | — | 86.22 | — | — | |
| PSO-MergingModel=FLAN-T52026.05 | 81.2 | — | — | 83.6 | 89.5 | 91.1 | 68.2 | 71.9 | 80.6 | 81.2 | — | 83.8 | — | — | |
| SSR2026.06 | 80.9 | — | — | 83.6 | 90.1 | 92.7 | 69.3 | 74.9 | 82.1 | 81.4 | — | 73.3 | — | — | |
| BinaryBERT#Bits=1/4, Size (MB)=16.5, Data Augmentation=true2026.04 | 80.8 | 83.9 | 84.2 | 91.2 | 91.4 | 93.7 | 53.3 | 88.6 | 86 | 71.5 | — | — | — | — | |
| Ours#Bits=1/1.5, Size (MB)=13.4, Data Augmentation=true2026.04 | 80.4 | 81 | 81.3 | 89.2 | 85.5 | 92.9 | 49.3 | 87.4 | 85.5 | 72.2 | — | — | — | — | |
| PrefixTuningBackbone=RoBERTaBase, Param. (M)=0.962025.05 | 80.18 | — | — | 87.75 | 90.77 | 92.17 | 55.31 | 88.48 | 87.25 | 54.51 | — | 85.21 | — | — | |
| DOGE TA2026.06 | 79.9 | — | — | 83.5 | 90.3 | 92.5 | 69.1 | 71.1 | 80.9 | 79.8 | — | 71.9 | — | — | |
| DELLAModel=FLAN-T52026.05 | 79.5 | — | — | 83.8 | 89.9 | 91.1 | 69.3 | 76 | 79.9 | 82 | — | 64.4 | — | — | |
| CMAModel=FLAN-T52026.05 | 79.4 | — | — | 73.9 | 89.4 | 92.2 | 70.9 | 69.7 | 75.7 | 80.9 | — | 82.9 | — | — | |
| AdaMergingModel=FLAN-T52026.05 | 79.4 | — | — | 81.7 | 89.8 | 91.4 | 69.9 | 66.1 | 79.9 | 79.1 | — | 77.2 | — | — | |
| TAModel=FLAN-T52026.05 | 78.9 | — | — | 83.9 | 89.8 | 91.7 | 69.1 | 73.2 | 79.4 | 81.2 | — | 62.7 | — | — | |
| DAREModel=FLAN-T52026.05 | 78.7 | — | — | 83.9 | 89.9 | 91.7 | 69.5 | 69.8 | 79.7 | 81.2 | — | 63.8 | — | — | |
| DARE-TIESModel=FLAN-T52026.05 | 78.6 | — | — | 83.8 | 89.8 | 91.3 | 69.3 | 71.1 | 79.7 | 81.6 | — | 62.5 | — | — | |
| Weight Averaging2026.06 | 78.2 | — | — | 83.8 | 90.1 | 91.2 | 69.7 | 72 | 78.9 | 80.5 | — | 59.7 | — | — | |
| BERT-base2026.06 | 78.15 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Concrete TA2026.06 | 78 | — | — | 83.5 | 89.9 | 91.6 | 69.1 | 73.4 | 78.4 | 79.4 | — | 58.1 | — | — | |
| TernaryBERT#Bits=2/8, Size (MB)=28.0, Data Augmentation=true2026.04 | 77.8 | 83.3 | 83.3 | 90.1 | 90 | 92.9 | 47.8 | 84.3 | 82.6 | 68.4 | — | — | — | — | |
| BiT#Bits=1/1, Size (MB)=13.4, Data Augmentation=true2026.04 | 77.5 | 79.5 | 79.4 | 85.4 | 82.5 | 92.3 | 38.2 | 84.2 | 83 | 69.7 | — | — | — | — | |
| Ties Merging2026.06 | 77.5 | — | — | 83.7 | 89.8 | 91.6 | 68.3 | 71.2 | 79.4 | 79.4 | — | 56.3 | — | — | |
| TIESModel=FLAN-T52026.05 | 77.4 | — | — | 83.4 | 89.3 | 91.3 | 69.2 | 68.4 | 77.7 | 80.5 | — | 59.4 | — | — | |
| Task Arithmetic2026.06 | 77.4 | — | — | 83.7 | 89.8 | 91.5 | 68.8 | 72.4 | 78.7 | 79.1 | — | 55.2 | — | — | |
| moBERTo-8ktokenizer=original2026.06 | 77.24 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| moBERTotokenizer=original2026.06 | 77.05 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BiPFT#Bits=1/2, Size (MB)=14.9, Data Augmentation=false2026.04 | 76.8 | 77 | 76.9 | 89 | 86.6 | 88.1 | 36.2 | 87 | 84.1 | 66.1 | — | — | — | — | |
| MLBERT#Bits=1/4, Data Augmentation=true2026.04 | 76.3 | — | — | — | — | 93.5 | 53.5 | — | 86.1 | 72.2 | — | — | — | — | |
| FisherModel=FLAN-T52026.05 | 76 | — | — | 83.6 | 84.6 | 88.1 | 69.3 | 74.4 | 76.7 | 77.6 | — | 54 | — | — | |
| RankMeanModel=FLAN-T52026.05 | 75.7 | — | — | 82.1 | 88.5 | 91.2 | 69.1 | 62.2 | 76.2 | 80.1 | — | 56.5 | — | — | |
| MLBERT#Bits=1/3, Size (MB)=13.0, Data Augmentation=true2026.04 | 75.1 | — | — | — | — | 93.4 | 52.1 | — | 85.8 | 69.2 | — | — | — | — | |
| PromptTuningBackbone=RoBERTaBase, Param. (M)=0.612025.05 | 74.93 | — | — | 82.99 | 80.03 | 91.09 | 49.37 | 82.44 | 74.83 | 58.12 | — | 80.57 | — | — | |
| BEBERT#Bits=1/1, Data Augmentation=true2026.04 | 74.7 | 75.7 | 76.6 | 84.9 | 80.7 | 91.8 | 42.6 | — | 82.4 | 65.8 | — | — | — | — | |
| NeoBERT-base2026.06 | 74.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MLBERT#Bits=1/2, Data Augmentation=true2026.04 | 74 | — | — | — | — | 92.6 | 50.2 | — | 85.2 | 68.1 | — | — | — | — | |
| moBERTo-SWM-8ktokenizer=Portuguese2026.06 | 73.54 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Qwen-base2026.06 | 72.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| moBERTo-SWMtokenizer=Portuguese2026.06 | 71.28 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Qwen3-0.6B-PT2026.06 | 70.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BERTimbau-base2026.06 | 67.72 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BiBERT#Bits=1/1, Size (MB)=13.4, Data Augmentation=true2026.04 | 67 | 66.1 | 67.5 | 84.8 | 76 | 90.9 | 37.8 | 56.7 | 78.8 | 61 | — | — | — | — | |
| NeoBERT-PT2026.06 | 66.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| RegMeanModel=FLAN-T52026.05 | 65.4 | — | — | 77.2 | 79.3 | 86 | 69.1 | 48.1 | 75.3 | 61.7 | — | 26.6 | — | — | |
| RFALearning Strategy=unlearnable2026.06 | 64.9 | — | — | 86.3 | 75.8 | 90.1 | 27.6 | 41.2 | 64.2 | 53 | — | 80.8 | — | — | |
| PerformerLearning Strategy=unlearnable2026.06 | 63.3 | — | — | 84.4 | 76.3 | 88.9 | 22.8 | 38.8 | 66.3 | 51.6 | — | 77.5 | — | — | |
| TernaryBERT#Bits=2/2, Size (MB)=28.0, Data Augmentation=true2026.04 | 58.3 | 40.3 | 40 | 63.1 | 50 | 87.5 | 20.6 | 72.5 | 72 | 47.2 | — | — | — | — | |
| BinaryBERT#Bits=1/2, Size (MB)=16.5, Data Augmentation=true2026.04 | 52 | 62.7 | 63.9 | 79.9 | 51 | 89.6 | 33 | 11.4 | 71 | 55.9 | — | — | — | — | |
| BinaryBERT#Bits=1/1, Size (MB)=16.5, Data Augmentation=true2026.04 | 50.1 | 35.6 | 35.3 | 66.2 | 66.1 | 78.3 | 7.3 | 22.1 | 69.3 | 57.7 | — | — | — | — | |
| TernaryBERT#Bits=2/1, Size (MB)=28.0, Data Augmentation=true2026.04 | 45.5 | 32.7 | 33 | 74.1 | 50.9 | 80.3 | 6.5 | 10.3 | 71.5 | 53.4 | — | — | — | — | |
| AdaCubicBackbone=SqueezeBERT, Epochs=15, Evaluation Protocol=Fine-tuned2026.04 | — | 81.88 | 81.89 | 87.23 | 90.01 | 90.71 | — | 88.32 | 90.42 | 70.39 | 56.33 | — | — | — | |
| AdaHessianBackbone=SqueezeBERT, Epochs=15, Evaluation Protocol=Fine-tuned2026.04 | — | 81.65 | 81.57 | 87.42 | 89.47 | 90.71 | — | 87.86 | 85.62 | 64.98 | 56.33 | — | — | — | |
| BERT-Large fine-tunedBackbone=BERT-Large, Fine-tuned=true, Secure Multi-Party Computation (MPC)=true2026.04 | — | — | — | 88.08 | 92.24 | 93.46 | 63.35 | 90.36 | 91.62 | 75.45 | — | 86.64 | — | 1 | |
| SecureRouter + MPC optimized model poolMPC Router=MLP, MPC optimized model pool=true, Secure Multi-Party Computation (MPC)=true2026.04 | — | — | — | 88.05 | 92 | 93.1 | 59.1 | 89.02 | 91.78 | 75.09 | — | 86.73 | — | 1.53 | |
| SGDBackbone=SqueezeBERT, Epochs=15, Evaluation Protocol=Fine-tuned2026.04 | — | 82.45 | 82.05 | 87.75 | 90.37 | 91.62 | — | 88.63 | 90.94 | 70.39 | 56.33 | — | — | — |