Natural Language Understanding on GLUE original (val)
92.4QQPFFT
Evaluation results
| Method | Links | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| FFTBackbone=DeBERTaV3-base, #Params=184M, Mem.=5.9, Train batch size=32, Max seq. len.=128, #Epochs=102026.05 | 92.4 | — | — | — | — | — | — | 91.15 | 89.9 | |
| LOFT P_prin, T_free r=46Backbone=DeBERTaV3-base, #Params=0.15M, Mem.=4.1, Train batch size=32, Max seq. len.=128, #Epochs=102026.05 | 92.34 | — | — | — | — | — | — | 91.38 | 90.42 | |
| PiSSA r=8Backbone=DeBERTaV3-base, #Params=1.33M, Mem.=4.5, Train batch size=32, Max seq. len.=128, #Epochs=102026.05 | 92.33 | — | — | — | — | — | — | 91.35 | 90.37 | |
| LOFT P_skew r=46Backbone=DeBERTaV3-base, #Params=0.07M, Mem.=4.1, Train batch size=32, Max seq. len.=128, #Epochs=102026.05 | 92.13 | — | — | — | — | — | — | 91.24 | 90.35 | |
| BOFT b=8 m=2Backbone=DeBERTaV3-base, #Params=1.41M, Mem.=6.3, Train batch size=32, Max seq. len.=128, #Epochs=102026.05 | 92.1 | — | — | — | — | — | — | 91.18 | 90.25 | |
| OFTv2 b=16Backbone=DeBERTaV3-base, #Params=1.29M, Mem.=4.5, Train batch size=32, Max seq. len.=128, #Epochs=102026.05 | 92.1 | — | — | — | — | — | — | 91.22 | 90.33 | |
| DoRA r=16Backbone=DeBERTaV3-base, #Params=1.41M, Mem.=5.8, Train batch size=32, Max seq. len.=128, #Epochs=102026.05 | 92.1 | — | — | — | — | — | — | 91.2 | 90.29 | |
| LoRA r=8Backbone=DeBERTaV3-base, #Params=1.33M, Mem.=4.5, Train batch size=32, Max seq. len.=128, #Epochs=102026.05 | 91.99 | — | — | — | — | — | — | 91.32 | 90.65 | |
| FRoD(s=0.02)Backbone=RoBERTa-large, # Params (%)=2.492025.12 | 91.63 | 86.86 | 96.33 | 90.44 | 89.58 | 94.4 | 84.48 | 90.53 | — | |
| Full FTBackbone=RoBERTa-large, # Params (%)=1002025.12 | 91.58 | 84.27 | 96.44 | 90.44 | 90.84 | 94.58 | 84.84 | 90.42 | — | |
| GOATBackbone=RoBERTa-large, # Params (%)=4.502025.12 | 91.4 | 86.86 | 96.21 | 84.55 | 89.55 | 94.19 | 85.56 | 89.76 | — | |
| qGOFTv2Backbone=DeBERTaV3-base, #Params=0.33M, Mem.=18.5, Train batch size=32, Max seq. len.=128, #Epochs=102026.05 | 91.34 | — | — | — | — | — | — | 90.76 | 90.17 | |
| MiLoRABackbone=RoBERTa-large, # Params (%)=4.002025.12 | 91.33 | 84.65 | 96.1 | 86.02 | 89.51 | 94.12 | 84.83 | 89.51 | — | |
| PiSSABackbone=RoBERTa-large, # Params (%)=4.002025.12 | 91.24 | 69.12 | 95.98 | 82.84 | 88.94 | 93.59 | 73.29 | 85 | — | |
| DoRABackbone=RoBERTa-large, # Params (%)=4.002025.12 | 91.24 | 85.33 | 95.99 | 84.07 | 89.52 | 93.54 | 84.48 | 89.17 | — | |
| LOFT P_prin r=46Backbone=DeBERTaV3-base, #Params=0.07M, Mem.=4.1, Train batch size=32, Max seq. len.=128, #Epochs=102026.05 | 91.05 | — | — | — | — | — | — | 90.58 | 90.12 | |
| PSOFT r=46Backbone=DeBERTaV3-base, #Params=0.08M, Mem.=4.1, Train batch size=32, Max seq. len.=128, #Epochs=102026.05 | 91 | — | — | — | — | — | — | 90.42 | 89.83 | |
| GOFTv2Backbone=DeBERTaV3-base, #Params=0.08M, Mem.=18.5, Train batch size=32, Max seq. len.=128, #Epochs=102026.05 | 90.85 | — | — | — | — | — | — | 90.48 | 90.1 | |
| RandLoRABackbone=RoBERTa-large, # Params (%)=0.842025.12 | 90.24 | 84.66 | 96.1 | 89.71 | 89.58 | 94.25 | 80.14 | 89.24 | — | |
| LoRABackbone=RoBERTa-large, # Params (%)=4.002025.12 | 90.06 | 83.41 | 95.64 | 83.33 | 89 | 93.28 | 84.47 | 88.46 | — | |
| VeRABackbone=RoBERTa-large, # Params (%)=0.222025.12 | 89.3 | 81.78 | 95.76 | 86.59 | 89.48 | 94.39 | 75.09 | 87.48 | — |