Natural Language Understanding on GLUE (CoLA, MRPC, QNLI, RTE, STS-B, SST-2)
96.5SST-2 AccuracyRandLoRA
Evaluation Results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| RandLoRABackbone=RoBERTa-Large, Params (M)=1.78, Rank (r)=1002026.04 | 96.5 | 90.1 | 67.7 | 94.7 | 85.3 | 92.3 | 87.8 | |
| Full Fine-TuningBackbone=RoBERTa-Large, Params (M)=354.312026.04 | 96.4 | 89.5 | 67.5 | 94.7 | 84.7 | 92.1 | 87.5 | |
| LoRABackbone=RoBERTa-Large, Params (M)=1.77, Rank (r)=42026.04 | 96.2 | 89.7 | 66.6 | 94.7 | 85.2 | 92.2 | 87.4 | |
| GiVABackbone=RoBERTa-Large, Params (M)=0.22, Rank (r)=8, Initialization strategy=TopSV-B2026.04 | 96.2 | 90.3 | 67.5 | 94.7 | 84.5 | 91.6 | 87.5 | |
| VeRABackbone=RoBERTa-Large, Params (M)=0.26, Rank (r)=2562026.04 | 96.1 | 89.8 | 68.6 | 94.8 | 83.2 | 92 | 87.4 | |
| GiVABackbone=RoBERTa-Large, Params (M)=0.22, Rank (r)=8, Initialization strategy=SecSV-B2026.04 | 96.1 | 89.4 | 68 | 95 | 82.3 | 91.6 | 87.1 | |
| GiVABackbone=RoBERTa-Large, Params (M)=0.22, Rank (r)=8, Initialization strategy=Rand-B2026.04 | 96.1 | 89.8 | 67 | 94.7 | 85.3 | 92 | 87.5 | |
| RandLoRABackbone=RoBERTa-Base, Params (M)=0.72, Rank (r)=642026.04 | 94.9 | 88.8 | 62.6 | 92.8 | 74.8 | 90.6 | 84.1 | |
| Full Fine-TuningBackbone=RoBERTa-Base, Params (M)=124.062026.04 | 94.8 | 89.6 | 62.1 | 92.8 | 75.8 | 90.9 | 84.3 | |
| LoRABackbone=RoBERTa-Base, Params (M)=0.66, Rank (r)=42026.04 | 94.7 | 87.7 | 59.2 | 92.7 | 74.8 | 90.5 | 83.3 | |
| VeRABackbone=RoBERTa-Base, Params (M)=0.16, Rank (r)=10242026.04 | 94.5 | 88.2 | 62.3 | 92.5 | 73.6 | 90.6 | 83.6 | |
| GiVABackbone=RoBERTa-Base, Params (M)=0.08, Rank (r)=8, Initialization strategy=Rand-B2026.04 | 94.5 | 89.3 | 61.6 | 92.3 | 74.8 | 90.5 | 83.8 | |
| GiVABackbone=RoBERTa-Base, Params (M)=0.08, Rank (r)=8, Initialization strategy=TopSV-B2026.04 | 94.2 | 88.7 | 59.1 | 92.3 | 75.5 | 90.4 | 83.4 | |
| GiVABackbone=RoBERTa-Base, Params (M)=0.08, Rank (r)=8, Initialization strategy=SecSV-B2026.04 | 94.2 | 89.4 | 60 | 92.2 | 73.4 | 90.5 | 83.3 |