Natural Language Understanding on GLUE (CoLA, MRPC, RTE Subset)
91RTE AccuracyAdaPreLoRA AdamW
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| AdaPreLoRA AdamWModel=Qwen2-7B, Rank=82026.05 | 91 | — | 90.4 | |
| AdamWModel=Qwen2-7B, Rank=82026.05 | 90.6 | — | 90 | |
| Scaled AdamWModel=Qwen2-7B, Rank=82026.05 | 90.6 | — | 87 | |
| AdaPreLoRA AdamWModel=Mistral-7B, Rank=82026.05 | 89.5 | 71.4 | 90 | |
| AdamWModel=Mistral-7B, Rank=82026.05 | 89.4 | 69.4 | 89.5 | |
| Scaled AdamWModel=Mistral-7B, Rank=82026.05 | 89.1 | 71.5 | 89.7 | |
| SOAPModel=Mistral-7B, Rank=82026.05 | 88.9 | 68.3 | 86.3 | |
| LoRA-Pro AdamWModel=Mistral-7B, Rank=82026.05 | 88.8 | 68.5 | 84.8 | |
| LoRA-Pro AdamWModel=Qwen2-7B, Rank=82026.05 | 88.1 | — | 89.2 | |
| SOAPModel=Qwen2-7B, Rank=82026.05 | 86.6 | — | 89.2 |