Natural Language Understanding on GLUE (SST2, MRPC, QNLI, COLA, RTE, STSB Subset)
96.2Accuracy (SST-2)Full-FT
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| Full-FTModel=RoBERTa-large, Batch size=32, LoRA rank=N/A, Sequence length=1282023.08 | 96.2 | 90.1 | 80 | 94.3 | 86 | 92.1 | 89.8 | |
| LoRA-QVModel=RoBERTa-large, Batch size=32, LoRA rank=8, Sequence length=1282023.08 | 96.2 | 90.3 | 72 | 94.8 | 85.2 | 92.3 | 88.5 | |
| LoRA-FAModel=RoBERTa-large, Batch size=32, LoRA rank=8, Sequence length=1282023.08 | 96.1 | 91.2 | 80.9 | 94.4 | 85.5 | 92 | 90 | |
| VeRAModel=RoBERTa-large, Batch size=32, LoRA rank=8, Sequence length=1282023.08 | 96 | 90.8 | 70 | 94.4 | 85.9 | 91.6 | 88.1 | |
| PiSSAModel=RoBERTa-large, Batch size=32, LoRA rank=8, Sequence length=1282023.08 | 96 | 89.9 | 73.9 | 94.1 | 85 | 89.9 | 88.1 | |
| LoRA+Model=RoBERTa-large, Batch size=32, LoRA rank=8, Sequence length=1282023.08 | 95.9 | 90.9 | 77 | 94.1 | 84 | 91.9 | 89 | |
| LoRAModel=RoBERTa-large, Batch size=32, LoRA rank=8, Sequence length=1282023.08 | 95.2 | 89.3 | 72 | 94.5 | 82.4 | 92 | 87.6 | |
| LoRA-GAModel=RoBERTa-large, Batch size=32, LoRA rank=8, Sequence length=1282023.08 | 95.1 | 90.9 | 80 | 94 | 84.1 | 92.1 | 89.4 | |
| LoRA-QVModel=RoBERTa-base, Batch size=64, LoRA rank=8, Sequence length=1282023.08 | 95 | 89.6 | 70.2 | 93.3 | 76.8 | 91.4 | 86.1 | |
| AdaLoRAModel=RoBERTa-large, Batch size=32, LoRA rank=8, Sequence length=1282023.08 | 94.9 | 88.9 | 71.9 | 93 | 81.9 | 90 | 86.8 | |
| Full-FTModel=RoBERTa-base, Batch size=64, LoRA rank=N/A, Sequence length=1282023.08 | 94.7 | 90 | 79.1 | 92.3 | 77.1 | 90.7 | 87.3 | |
| PiSSAModel=RoBERTa-base, Batch size=64, LoRA rank=8, Sequence length=1282023.08 | 94.3 | 84.7 | 73 | 92.9 | 77 | 89.9 | 85.3 | |
| LoRA-GAModel=RoBERTa-base, Batch size=64, LoRA rank=8, Sequence length=1282023.08 | 94.3 | 87.8 | 80 | 93.1 | 77 | 88.1 | 86.7 | |
| LoRA-FAModel=RoBERTa-base, Batch size=64, LoRA rank=8, Sequence length=1282023.08 | 94.1 | 90.4 | 79.9 | 92.3 | 77.6 | 91.1 | 87.5 | |
| QLoRAModel=RoBERTa-large, Batch size=32, LoRA rank=8, Sequence length=1282023.08 | 94.1 | 87 | 69 | 90.5 | 71.1 | 89.9 | 83.6 | |
| LoRAModel=RoBERTa-base, Batch size=64, LoRA rank=8, Sequence length=1282023.08 | 94 | 83.3 | 69 | 81.6 | 74.8 | 88.8 | 81.9 | |
| DoRAModel=RoBERTa-base, Batch size=64, LoRA rank=8, Sequence length=1282023.08 | 94 | 83.6 | 71.1 | 92.9 | 75.1 | 87.2 | 84 | |
| DoRAModel=RoBERTa-large, Batch size=32, LoRA rank=8, Sequence length=1282023.08 | 94 | 89 | 71.9 | 93 | 83 | 91 | 87 | |
| LoRA+Model=RoBERTa-base, Batch size=64, LoRA rank=8, Sequence length=1282023.08 | 93.9 | 85.2 | 75.1 | 93.1 | 77 | 87.1 | 85.2 | |
| AdaLoRAModel=RoBERTa-base, Batch size=64, LoRA rank=8, Sequence length=1282023.08 | 93.5 | 82.3 | 70.1 | 92 | 73.1 | 87.1 | 83 | |
| VeRAModel=RoBERTa-base, Batch size=64, LoRA rank=8, Sequence length=1282023.08 | 92.5 | 89.4 | 67.2 | 91.7 | 76.9 | 89.3 | 84.5 | |
| QLoRAModel=RoBERTa-base, Batch size=64, LoRA rank=8, Sequence length=1282023.08 | 92.2 | 82.9 | 65 | 82.4 | 56.1 | 83.2 | 77.1 |