Natural Language Understanding on GLUE (Full Suite Performance)
97.5SST-2 AccLoRA
Evaluation results
| Method | Links | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| LoRAModel=DeBERTa-XXL, Trainable Param. (M)=9.4, Rank=322025.12 | 97.5 | 92 | 91.2 | 68.7 | 96 | 91.9 | 92.8 | 92.4 | 90.3 | |
| LoRA+Model=DeBERTa-XXL, Trainable Param. (M)=9.4, Rank=322025.12 | 97.5 | 91.7 | 91.2 | 68.7 | 96 | 92.3 | 94.6 | 92.4 | 90.5 | |
| FFTModel=DeBERTa-XXL, Trainable Param. (M)=15002025.12 | 97.2 | 91.8 | 92 | 72 | 96 | 92.7 | 93.9 | 92.9 | 91.1 | |
| Dual LoRAModel=DeBERTa-XXL, Trainable Param. (M)=9.4, Rank=162025.12 | 97.1 | 91.9 | 91.9 | 74 | 96.2 | 92.6 | 95.3 | 93.4 | 91.6 | |
| DoRAModel=DeBERTa-XXL, Trainable Param. (M)=9.4, Rank=322025.12 | 96.9 | 91.9 | 90.9 | 71.2 | 95.8 | 92.3 | 92.6 | 92.3 | 90.5 | |
| LoRAModel=DeBERTa-XXL, Trainable Param. (M)=4.7, Rank=162025.12 | 96.6 | 91.7 | 89.7 | 70.8 | 95.7 | 92.6 | 95 | 92.4 | 90.6 | |
| FFTModel=RoBERTa-large, Trainable Param. (M)=3552025.12 | 96.4 | 90.2 | 90.9 | 68 | 94.7 | 92.2 | 86.6 | 92.4 | 88.9 | |
| Dual LoRAModel=RoBERTa-large, Trainable Param. (M)=1.6, Rank=82025.12 | 96.4 | 90.5 | 91.9 | — | 95.1 | 91.2 | 89.5 | 92.6 | 89.7 | |
| LoRA+Model=RoBERTa-large, Trainable Param. (M)=1.6, Rank=162025.12 | 96.3 | 90.3 | 91.4 | 68.7 | 94.7 | 91.6 | 88.8 | 92.5 | 89.3 | |
| GaRareModel=RoBERTa-large, Trainable Param. (M)=1.6, Rank=162025.12 | 96.2 | 91.3 | 91.7 | 67.9 | 94.6 | 91.8 | 87.4 | 92.3 | 89.2 | |
| DoRAModel=RoBERTa-large, Trainable Param. (M)=1.6, Rank=162025.12 | 96.2 | 90.5 | 89.7 | 68.5 | 92.6 | 91.5 | 89.2 | 92.3 | 88.8 | |
| GaLoreModel=RoBERTa-large, Trainable Param. (M)=1.6, Rank=162025.12 | 96.1 | 90.8 | 91.7 | 68.3 | 95.7 | 91.9 | 87 | 92.5 | 89.3 | |
| LoRAModel=RoBERTa-large, Trainable Param. (M)=1.6, Rank=162025.12 | 95.9 | 90.2 | 90.9 | 66 | 94.4 | 91.6 | 87.4 | 92.3 | 88.6 | |
| Dual LoRAModel=RoBERTa-base, Trainable Param. (M)=0.6, Rank=82025.12 | 95.8 | 87.8 | 91.7 | 67.8 | 93.3 | 90.7 | 88.1 | 91.7 | 88.3 | |
| VeRAModel=RoBERTa-large, Trainable Param. (M)=0.32025.12 | 95.8 | — | 89.3 | 65.3 | 94.1 | — | 81.6 | 91.8 | — | |
| LoRAModel=RoBERTa-large, Trainable Param. (M)=0.8, Rank=82025.12 | 95.6 | 90.2 | 89.5 | 63.8 | 94.5 | 91.5 | 88.8 | 92.5 | 88.3 | |
| RandLoRAModel=RoBERTa-large, Trainable Param. (M)=1.82025.12 | 95.5 | — | 90.1 | 67.4 | 94.1 | — | 84.5 | 91.4 | — | |
| DoRAModel=RoBERTa-base, Trainable Param. (M)=0.6, Rank=162025.12 | 95.3 | 87.7 | 87.8 | 64.8 | 92.6 | 90.8 | 82.2 | 90.8 | 86.5 | |
| LoRA+Model=RoBERTa-base, Trainable Param. (M)=0.6, Rank=162025.12 | 95.2 | 87.8 | 90.4 | 65.9 | 92.6 | 91.2 | 82.3 | 91.4 | 87.1 | |
| Delta-LoRAModel=RoBERTa-base, Trainable Param. (M)=0.3, Rank=82025.12 | 95.1 | 87.5 | 90.2 | 63.8 | 93.1 | 90.9 | 87 | 91.6 | 87.4 | |
| LoRAModel=RoBERTa-base, Trainable Param. (M)=0.6, Rank=162025.12 | 95.1 | 87 | 89 | 63.9 | 93 | 91.2 | 83.4 | 91.1 | 86.7 | |
| FFTModel=RoBERTa-base, Trainable Param. (M)=1252025.12 | 94.8 | 87.6 | 90.2 | 63.6 | 92.8 | 91.9 | 78.7 | 91.2 | 86.4 | |
| LoRAModel=RoBERTa-base, Trainable Param. (M)=0.3, Rank=82025.12 | 94.6 | 87 | 89.2 | 60.9 | 92.9 | 90.7 | 92 | 91.1 | 86.1 | |
| GaLoreModel=RoBERTa-base, Trainable Param. (M)=0.3, Rank=82025.12 | 94.4 | 87.2 | 92 | 61.8 | 92.3 | 91.2 | 79.1 | 90.8 | 85.9 | |
| GaRareModel=RoBERTa-base, Trainable Param. (M)=0.3, Rank=82025.12 | 94.4 | 87.2 | 91.5 | 61.1 | 92.3 | 90.9 | 79.3 | 90.3 | 85.9 | |
| CorDAModel=RoBERTa-base, Trainable Param. (M)=21, Rank=1282025.12 | 93.1 | — | 89.7 | 59.6 | 91.5 | — | 88.1 | 90.2 | — | |
| RandLoRAModel=RoBERTa-base, Trainable Param. (M)=0.72025.12 | 92.2 | — | 88 | 59.4 | 91.3 | — | 74.7 | 90.3 | — | |
| VeRAModel=RoBERTa-base, Trainable Param. (M)=0.32025.12 | 91.9 | — | 88.4 | 59.9 | 90.5 | — | 74.9 | 90.4 | — |