Natural Language Understanding on GLUE (CoLA, SST-2, MRPC, QQP, MNLI, QNLI, RTE Subset)
86.86CoLA AccuracyGOAT
Evaluation Results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| GOAT# Params (%)=4.50, Total rank=322025.02 | 86.86 | 96.21 | 84.55 | 91.4 | 89.55 | 94.19 | 85.56 | 89.76 | |
| Full FT MoE# Params (%)=6982025.02 | 86.02 | 96.22 | 85.05 | 92.2 | 90.2 | 95.1 | 84.48 | 89.9 | |
| DoRA# Params (%)=4.00, Total rank=322025.02 | 85.33 | 95.99 | 84.07 | 91.24 | 89.52 | 93.54 | 84.48 | 89.17 | |
| MiLoRA# Params (%)=4.00, Total rank=322025.02 | 84.65 | 96.1 | 86.02 | 91.33 | 89.51 | 94.12 | 84.83 | 89.51 | |
| Full FT# Params (%)=1002025.02 | 84.27 | 95.98 | 85.29 | 91.58 | 89.83 | 94.49 | 84.84 | 89.47 | |
| AdaMoLE# Params (%)=4.56, Total rank=322025.02 | 83.99 | 95.76 | 86.03 | 91.48 | 89.21 | 93.64 | 83.75 | 89.12 | |
| MoLoRA# Params (%)=4.50, Total rank=322025.02 | 83.94 | 96.1 | 87.75 | 91.45 | 89.36 | 93.9 | 84.11 | 89.52 | |
| HydraLoRA# Params (%)=2.75, Total rank=322025.02 | 83.89 | 95.52 | 85.04 | 91.02 | 89.34 | 93.87 | 81.22 | 88.56 | |
| rsLoRA# Params (%)=4.00, Total rank=322025.02 | 83.51 | 95.98 | 86.02 | 90.75 | 88.97 | 93.84 | 84.12 | 89.03 | |
| LoRA# Params (%)=4.00, Total rank=322025.02 | 83.41 | 95.64 | 83.33 | 90.06 | 89 | 93.28 | 84.47 | 88.46 | |
| PiSSA# Params (%)=4.00, Total rank=322025.02 | 69.12 | 95.98 | 82.84 | 91.24 | 88.94 | 93.59 | 73.29 | 85 |