Natural Language Understanding on GLUE (val) - Subset (CoLA, MNLI, MRPC, QNLI, QQP, RTE, SST-2, STS-B)
82.4CoLA ScoreProbSurgery
Evaluation results
| Method | Links | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| ProbSurgeryModel=FLAN-T5-Base, Reference=Task Arithmetic2026.05 | 82.4 | 69.1 | 78.3 | 80.6 | 89.8 | 83.4 | 81.2 | 92.5 | 82.2 | |
| Fine-tuned (STL)Model=FLAN-T5-Large2026.05 | 80.2 | 88.5 | 89.2 | 94.4 | 87.2 | 91.7 | 95.2 | 90.9 | 89.6 | |
| IndividualBase Model=Flan-T5-large, Fine-tuning Method=LoRA, Merging Strategy Category=Basic2023.12 | 80.2 | 88.5 | 89.2 | 94.4 | 87.2 | 91.7 | 95.2 | 90.9 | 89.6 | |
| FEATCALModel=FLAN-T5-Large, Reference=Task Arithmetic2026.05 | 79.2 | 87.8 | 89 | 93.9 | 86.9 | 88.4 | 95.3 | 90.8 | 88.9 | |
| ProbSurgeryModel=FLAN-T5-Large, Reference=Task Arithmetic2026.05 | 77.2 | 87.6 | 85 | 93.8 | 86 | 88.4 | 95.2 | 87.8 | 87.6 | |
| Ties-MergingBase Model=Flan-T5-large, Fine-tuning Method=LoRA, Merging Strategy Category=TA-Based2023.12 | 77.1 | 85.1 | 86.3 | 93.9 | 86 | 87.7 | 95.1 | 88 | 87.4 | |
| Task ArithmeticBase Model=Flan-T5-large, Fine-tuning Method=LoRA, Merging Strategy Category=TA-Based2023.12 | 76.9 | 85.4 | 85.3 | 93.9 | 85.8 | 88.1 | 95.2 | 87.8 | 87.3 | |
| Task ArithmeticModel=FLAN-T5-Large2026.05 | 76.8 | 85.4 | 85.3 | 94 | 85.8 | 88.1 | 95.2 | 87.7 | 87.3 | |
| LW AMBase Model=Flan-T5-large, Fine-tuning Method=LoRA, Merging Strategy Category=LW AM-Based2023.12 | 76.7 | 87.6 | 84.8 | 93.8 | 85.9 | 88.1 | 95.2 | 88.6 | 87.6 | |
| Concrete TABase Model=Flan-T5-large, Fine-tuning Method=LoRA, Merging Strategy Category=TA-Based2023.12 | 76.6 | 86.4 | 86 | 93.9 | 85.9 | 88.4 | 95.2 | 87.9 | 87.5 | |
| LW Concrete AMBase Model=Flan-T5-large, Fine-tuning Method=LoRA, Merging Strategy Category=LW AM-Based2023.12 | 76.1 | 87.7 | 85.5 | 93.8 | 85.9 | 88.1 | 95.4 | 87.1 | 87.5 | |
| SurgeryModel=FLAN-T5-Large, Reference=Task Arithmetic2026.05 | 76 | 87.9 | 86 | 93.9 | 86.2 | 89.9 | 95.2 | 89 | 88 | |
| Fine-tuned (STL)Model=FLAN-T5-Base2026.05 | 75 | 83.4 | 87.5 | 91.5 | 85.4 | 85.9 | 93.6 | 88.7 | 86.4 | |
| Weight AveragingBase Model=Flan-T5-large, Fine-tuning Method=LoRA, Merging Strategy Category=Basic2023.12 | 74.6 | 84.3 | 84.1 | 92.8 | 86.3 | 87.4 | 94.8 | 88 | 86.5 | |
| Pre-trainedModel=FLAN-T5-Large2026.05 | 73.7 | 56.6 | 82.4 | 91.1 | 85.5 | 85.6 | 94.3 | 87.5 | 82.1 | |
| Pre-trainedBase Model=Flan-T5-large, Fine-tuning Method=None, Merging Strategy Category=Basic2023.12 | 73.7 | 56.6 | 82.4 | 91.1 | 85.5 | 85.6 | 94.3 | 87.5 | 82.1 | |
| FEATCALModel=FLAN-T5-Base, Reference=Task Arithmetic2026.05 | 72.2 | 82.6 | 85.1 | 91.1 | 84.7 | 85.2 | 93 | 87.7 | 85.2 | |
| SurgeryModel=FLAN-T5-Base, Reference=Task Arithmetic2026.05 | 70.8 | 82.4 | 82.4 | 89.8 | 84.2 | 83 | 92.1 | 85.2 | 83.7 | |
| Task ArithmeticModel=FLAN-T5-Base2026.05 | 70.5 | 57.8 | 78.4 | 90.2 | 83.6 | 80.5 | 92.3 | 77.8 | 78.9 | |
| Pre-trainedModel=FLAN-T5-Base2026.05 | 69.1 | 56.5 | 76.2 | 88.4 | 82.1 | 80.1 | 91.2 | 62.2 | 75.7 | |
| BOFTConfig=r=2,b=4, #Params=0.746M2026.05 | 68.2 | 90.27 | 87.07 | 94.07 | 89.73 | 81.43 | 96 | 91.13 | 87.24 | |
| FTBackbone=RoBERTalarge, Parameters (×10^3)=355k, Rank=-2025.06 | 68 | 90.2 | 91 | 94.7 | 92.2 | 87 | 96.4 | 92.4 | 89 | |
| LoRABackbone=RoBERTalarge, Parameters (×10^3)=786, Rank=82025.06 | 68 | 90.6 | 84 | 94.8 | 91.6 | 87 | 95.7 | 91.9 | 88 | |
| OFTConfig=b=24, #Params=0.954M2026.05 | 67.63 | 90.27 | 87.93 | 94.3 | 89.77 | 84.27 | 95.83 | 91 | 87.63 | |
| LoCOConfig=n=1,r=4, #Params=0.663M2026.05 | 65.56 | 90.2 | 88.28 | 94.36 | 90.08 | 86.1 | 96.14 | 91.28 | 87.75 | |
| VeRABackbone=RoBERTalarge, Parameters (×10^3)=61, Rank=2562025.06 | 64 | 88.8 | 89.4 | 93.1 | 87.62 | 83 | 95.1 | 91.5 | 86.6 | |
| MetaTT-4DBackbone=RoBERTalarge, Parameters (×10^3)=92, Rank=322025.06 | 64 | 90 | 90.1 | 94.4 | 76 | 84.8 | 95.3 | 92.2 | 86 | |
| MetaTT-4DBackbone=RoBERTalarge, Parameters (×10^3)=39, Rank=162025.06 | 62.8 | 89.6 | 88.6 | 93.8 | 88.5 | 84.2 | 95.2 | 91.8 | 86.8 | |
| LoTRBackbone=RoBERTabase, Parameters (×10^3)=321, Rank=882025.06 | 61.3 | 84.7 | 88 | 92 | 86.9 | 67 | 93.3 | 91 | 83 | |
| LoTRBackbone=RoBERTalarge, Parameters (×10^3)=328, Rank=642025.06 | 61.3 | 90.3 | 89 | 94.8 | 89.2 | 84 | 95.9 | 91.6 | 87 | |
| LoRABackbone=RoBERTabase, Parameters (×10^3)=295, Rank=82025.06 | 61.1 | 87.3 | 88 | 91.3 | 90.1 | 73 | 94.2 | 90.7 | 84.5 | |
| FTBackbone=RoBERTabase, Parameters (×10^3)=125k, Rank=-2025.06 | 61 | 87.6 | 89.3 | 92.6 | 91.9 | 79 | 94.1 | 90.4 | 85.7 | |
| LoTRBackbone=RoBERTabase, Parameters (×10^3)=276, Rank=802025.06 | 61 | 84.6 | 89 | 92.1 | 86.8 | 71 | 93.4 | 90.9 | 83.6 | |
| MetaTT-4DBackbone=RoBERTabase, Parameters (×10^3)=156, Rank=642025.06 | 61 | 85.9 | 88.9 | 77 | 88.5 | 77.5 | 72 | 90.1 | 80 | |
| LoRETTAadpBackbone=RoBERTalarge, Parameters (×10^3)=133, Rank=64, 52025.06 | 61 | 89.69 | 88.1 | 94.08 | 89.6 | 72 | 95.5 | — | 84.3 | |
| MetaTT-4DBackbone=RoBERTabase, Parameters (×10^3)=45, Rank=242025.06 | 59.7 | 85.5 | 88.6 | 91 | 87.8 | 74.2 | 92.3 | 89.9 | 83.6 | |
| LoRTABackbone=RoBERTalarge, Parameters (×10^3)=9.1, Rank=82025.06 | 58.9 | 88.4 | 87.3 | 94 | 88.1 | 66.6 | 95.3 | 91.1 | 83.7 | |
| AdamWFine-tuning mode=Full, Backbone=DistilBERT base2025.10 | 58.8 | 84.11 | 85.78 | 91.89 | 90.63 | 68.23 | 92.89 | 89.32 | 82.71 | |
| MetaTT-4DBackbone=RoBERTabase, Parameters (×10^3)=13, Rank=82025.06 | 58.8 | 84.2 | 87.6 | 90.4 | 86.9 | 72.9 | 92 | 89.1 | 82.7 | |
| LoRTABackbone=RoBERTabase, Parameters (×10^3)=55, Rank=642025.06 | 58.6 | 86.1 | 88 | 92.2 | 89 | 75 | 93.6 | 89.3 | 84 | |
| VeRABackbone=RoBERTabase, Parameters (×10^3)=43, Rank=10242025.06 | 58 | 81 | 87.2 | 89.6 | 85.87 | 73.4 | 92.2 | 88.7 | 82 | |
| LoTRBackbone=RoBERTabase, Parameters (×10^3)=100, Rank=402025.06 | 58 | 85.2 | 88 | 92.5 | 87.6 | 53 | 93.8 | 89.8 | 81 | |
| LoRETTAadpBackbone=RoBERTabase, Parameters (×10^3)=57, Rank=64, 52025.06 | 57.9 | 84.6 | 86.4 | 92 | 88 | 70.3 | 93.3 | — | 81.8 | |
| MuonFine-tuning mode=Full, Backbone=DistilBERT base2025.10 | 57.53 | 80.68 | 84.56 | 90.55 | 89.58 | 68.95 | 91.97 | 89.03 | 81.61 | |
| LoRTABackbone=RoBERTabase, Parameters (×10^3)=6.9, Rank=82025.06 | 55.9 | 84.1 | 86.9 | 91.1 | 86.7 | 70.2 | 93 | 86.6 | 81.8 | |
| RMSSpectralFine-tuning mode=LoRA, Backbone=DistilBERT base2025.10 | 53.27 | 77.76 | 85.05 | 86.82 | 85.85 | 65.34 | 89.68 | 83.44 | 78.4 | |
| AdaMuonFine-tuning mode=LoRA, Backbone=DistilBERT base2025.10 | 51.74 | 78.36 | 84.8 | 87.39 | 85.84 | 66.79 | 90.37 | 84.27 | 78.69 | |
| HRAConfig=r=8, #Params=0.663M2026.05 | 51.67 | 90.2 | 88.03 | 93.57 | 90.2 | 84.5 | 95.7 | 91.25 | 85.64 | |
| MuonFine-tuning mode=LoRA, Backbone=DistilBERT base2025.10 | 49.92 | 72.37 | 85.78 | 85.01 | 83.22 | 66.06 | 88.65 | 83.92 | 76.87 | |
| RMSSpectral-SANIAFine-tuning mode=LoRA, Backbone=DistilBERT base2025.10 | 49.9 | 77.8 | 84.8 | 87.81 | 85.71 | 62.09 | 90.6 | 84.22 | 77.87 | |
| RMSSpectralFine-tuning mode=Full, Backbone=DistilBERT base2025.10 | 48 | 81.16 | 78.19 | 87.94 | 88.01 | — | — | — | 76.66 | |
| AdaMuonFine-tuning mode=Full, Backbone=DistilBERT base2025.10 | 47.7 | 81.09 | 82.6 | 87.57 | 87.82 | — | — | — | 77.36 | |
| AdamWFine-tuning mode=LoRA, Backbone=DistilBERT base2025.10 | 47.59 | 73.52 | 83.09 | 85.67 | 84.98 | 63.9 | 89.56 | 84.73 | 76.63 | |
| RMSSpectral-SANIAFine-tuning mode=Full, Backbone=DistilBERT base2025.10 | 45.87 | 81.22 | 81.37 | 87.88 | 87.82 | — | — | — | 76.83 |