Natural Language Understanding on GLUE (Core Task Performance)
69.3COLA ScoreVB-LoRAall
Evaluation Results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| VB-LoRAallBackbone=RoBERTa-large, Parameters=0.033M, Modules=all linear modules2024.05 | 69.3 | 96.3 | 91.9 | 94.4 | 87.4 | 91.8 | 88.5 | |
| VeRAallBackbone=RoBERTa-large, Parameters=0.258M, Modules=all linear modules2024.05 | 68.5 | 96.6 | 90.9 | 94.4 | 85.9 | 92.2 | 88.1 | |
| LilyBackbone=RoBERTa Large, # Trainable Parameters=0.5M2024.07 | 68.4 | 95.6 | 90.9 | 94.8 | 88.4 | 91.9 | 88.4 | |
| VB-LoRAqvBackbone=RoBERTa-large, Parameters=0.024M, Modules=query and value only2024.05 | 68.3 | 96.1 | 91.4 | 94.7 | 86.6 | 91.8 | 88.2 | |
| LoRAqvBackbone=RoBERTa-large, Parameters=0.786M, Modules=query and value only2024.05 | 68.2 | 96.2 | 90.2 | 94.8 | 85.2 | 92.3 | 87.8 | |
| LoRABackbone=RoBERTa Large, # Trainable Parameters=0.8M2024.07 | 68.2 | 96.2 | 90.2 | 94.8 | 85.2 | 92.3 | 87.8 | |
| VeRAqvBackbone=RoBERTa-large, Parameters=0.061M, Modules=query and value only2024.05 | 68 | 96.1 | 90.9 | 94.4 | 85.9 | 91.7 | 87.8 | |
| FFTBackbone=RoBERTa Large, # Trainable Parameters=356M2024.07 | 68 | 96.4 | 90.9 | 94.7 | 86.6 | 92.4 | 88.2 | |
| Tied-LoRAallBackbone=RoBERTa-large, Parameters=0.239M, Modules=all linear modules2024.05 | 66.8 | 94.8 | 90 | 94.1 | 82.3 | 91.6 | 86.6 | |
| Adpt_HBackbone=RoBERTa Large, # Trainable Parameters=0.8M2024.07 | 66.3 | 96.3 | 87.7 | 94.7 | 72.9 | 91.5 | 84.9 | |
| LilyBackbone=RoBERTa Base, # Trainable Parameters=0.3M2024.07 | 66 | 95 | 90.2 | 92.5 | 81.6 | 90.8 | 86 | |
| VeRAqvBackbone=RoBERTa-base, Parameters=0.043M, Modules=query and value only2024.05 | 65.6 | 94.6 | 89.5 | 91.8 | 78.7 | 90.7 | 85.2 | |
| FLORABackbone=RoBERTa Base, # Trainable Parameters=0.3M2024.07 | 65.4 | 91.2 | 85.8 | 92.3 | 65 | 87.6 | 81.2 | |
| Tied-LoRAallBackbone=RoBERTa-base, Parameters=0.109M, Modules=all linear modules2024.05 | 64.7 | 94.7 | 88.5 | 92.4 | 76.5 | 90.3 | 84.5 | |
| Tied-LoRAqvBackbone=RoBERTa-large, Parameters=0.066M, Modules=query and value only2024.05 | 64.7 | 94.8 | 89.7 | 94.1 | 81.2 | 90.8 | 85.9 | |
| VeRAallBackbone=RoBERTa-base, Parameters=0.157M, Modules=all linear modules2024.05 | 64.5 | 95.1 | 88.7 | 92.3 | 81.9 | 90.2 | 85.5 | |
| VB-LoRAallBackbone=RoBERTa-base, Parameters=0.027M, Modules=all linear modules2024.05 | 64.3 | 95 | 89.7 | 92.3 | 82.3 | 90.7 | 85.7 | |
| FTBackbone=RoBERTa-base, Parameters=125M2024.05 | 63.6 | 94.8 | 90.2 | 92.8 | 78.7 | 91.2 | 85.2 | |
| FFTBackbone=RoBERTa Base, # Trainable Parameters=125M2024.07 | 63.6 | 94.8 | 90.2 | 92.8 | 78.7 | 91.2 | 85.2 | |
| LoRAqvBackbone=RoBERTa-base, Parameters=0.295M, Modules=query and value only2024.05 | 63.4 | 95.1 | 89.7 | 93.3 | 86.6 | 91.5 | 86.6 | |
| VB-LoRAqvBackbone=RoBERTa-base, Parameters=0.023M, Modules=query and value only2024.05 | 63.3 | 94.4 | 89.5 | 92.2 | 82.3 | 90.8 | 85.4 | |
| LoRABackbone=RoBERTa Base, # Trainable Parameters=0.3M2024.07 | 63.3 | 94.8 | 89.8 | 92.9 | 78.2 | 91.5 | 85.1 | |
| Adpt_DBackbone=RoBERTa Base, # Trainable Parameters=0.9M2024.07 | 62.6 | 94.7 | 88.4 | 93 | 75.9 | 90.3 | 84.2 | |
| BitFitBackbone=RoBERTa Base, # Trainable Parameters=0.1M2024.07 | 62 | 93.7 | 92.7 | 91.8 | 81.5 | 90.8 | 85.4 | |
| AdaLoRABackbone=RoBERTa Base, # Trainable Parameters=0.3M2024.07 | 62 | 94.5 | 88.7 | 93.1 | 81 | 90.5 | 85 | |
| Tied-LoRAqvBackbone=RoBERTa-base, Parameters=0.043M, Modules=query and value only2024.05 | 61.9 | 94.4 | 88.5 | 92 | 76.2 | 89.8 | 83.8 | |
| DyLoRABackbone=RoBERTa Base, # Trainable Parameters=0.3M2024.07 | 61.1 | 94.3 | 89.5 | 92.2 | 78.7 | 91.1 | 84.5 | |
| Adpt_DBackbone=RoBERTa Base, # Trainable Parameters=0.3M2024.07 | 60.8 | 94.2 | 88.5 | 93.1 | 71.5 | 89.7 | 83 | |
| Upper BoundParams=125M, Size (MB)=5002026.02 | 59.91 | 91.28 | 89.01 | 92.31 | 79.7 | 90.9 | 83.9 | |
| Share-fullParams=0.012M, Size (MB)=0.292026.02 | 59.81 | 93.39 | 86.99 | 92.66 | 77.62 | 90.8 | 83.44 | |
| CorDA#Params=21M, Backbone=RoBERTa-base, Rank=128, Adaptation Mode=instruction-previewed2024.06 | 59.6 | 93.12 | 89.71 | 91.49 | 76.17 | 90.17 | 83.38 | |
| Full fine-tuning#Params=125M, Backbone=RoBERTa-base2024.06 | 59.56 | 93.81 | 88.48 | 92.07 | 74.01 | 90.49 | 83.07 | |
| LoRA (non-CL)Params=1.2M×6, Size (MB)=81.62026.02 | 59.56 | 93.35 | 86.76 | 92.53 | 77.61 | 90.81 | 83.43 | |
| Share (CL)Params=0.012M, Size (MB)=0.29, Task=T-22026.02 | 56.5 | — | 68.38 | — | 73.29 | — | — | |
| Share (CL)Params=0.012M, Size (MB)=0.29, Task=T-32026.02 | 56.5 | — | 68.38 | — | 73.29 | 88.91 | — | |
| Share (CL)Params=0.012M, Size (MB)=0.29, Task=T-42026.02 | 56.24 | — | 68.38 | 91.84 | 73.65 | 88.91 | — | |
| Share (CL)Params=0.012M, Size (MB)=0.29, Task=T-02026.02 | 56 | — | — | — | — | — | — | |
| Share (CL)Params=0.012M, Size (MB)=0.29, Task=T-52026.02 | 55.99 | 93.58 | 68.38 | 91.95 | 73.29 | 88.91 | 78.69 | |
| Share (CL)Params=0.012M, Size (MB)=0.29, Task=T-12026.02 | 55.54 | — | 68.38 | — | — | — | — | |
| LoRA#Params=21M, Backbone=RoBERTa-base, Rank=1282024.06 | 54.24 | 94.15 | 82.84 | 92.48 | 64.26 | 88.58 | 79.43 | |
| DoRA#Params=21M, Backbone=RoBERTa-base, Rank=1282024.06 | 51.93 | 93.58 | 83.58 | 92.59 | 64.98 | 88.71 | 79.23 |