NLG Evaluation Suite (GSM8K, MATH, HumanEval, MBPP, Commonsense)
60.34GSM8KFull Fine-Tuning
Evaluation Results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| Full Fine-TuningRank (r)=N/A, Backbone=Llama 2-7B, Learning Rate=2 x 10-52026.06 | 60.34 | 11.74 | 32.3 | 39.27 | 79.2 | 44.57 | |
| LoRA-α*Rank (r)=128, Backbone=Llama 2-7B, Learning Rate=2 x 10-52026.06 | 59.97 | 11.74 | 33.5 | 39.9 | 79.26 | 44.87 | |
| LoRA-η*Rank (r)=128, Backbone=Llama 2-7B, Learning Rate=2 x 10-52026.06 | 58.83 | 10.76 | 28.7 | 36.5 | 78.67 | 42.69 | |
| LoRA-α*Rank (r)=16, Backbone=Llama 2-7B, Learning Rate=2 x 10-52026.06 | 55.34 | 9.84 | 28.7 | 35.4 | 78.16 | 41.49 | |
| LoRA-η*Rank (r)=16, Backbone=Llama 2-7B, Learning Rate=2 x 10-52026.06 | 54.28 | 9.5 | 24.4 | 34.7 | 78.19 | 40.21 | |
| LoRA-αRank (r)=128, Backbone=Llama 2-7B, Learning Rate=2 x 10-52026.06 | 53.15 | 8.3 | 23.2 | 33.9 | 78.38 | 39.39 | |
| PiSSARank (r)=128, Backbone=Llama 2-7B, Learning Rate=2 x 10-52026.06 | 51.48 | 7.04 | 21.62 | 31.07 | 77.28 | 37.7 | |
| LoRAMRank (r)=128, Backbone=Llama 2-7B, Learning Rate=2 x 10-52026.06 | 51.12 | 7.25 | 22.03 | 31.53 | 77.81 | 37.95 | |
| RsLoRARank (r)=128, Backbone=Llama 2-7B, Learning Rate=2 x 10-52026.06 | 50.38 | 7.32 | 21.32 | 30.73 | 77.01 | 37.35 | |
| LoRA-αRank (r)=16, Backbone=Llama 2-7B, Learning Rate=2 x 10-52026.06 | 47.61 | 6.84 | 18.9 | 28.3 | 77.42 | 35.81 | |
| LoRA+Rank (r)=128, Backbone=Llama 2-7B, Learning Rate=2 x 10-52026.06 | 40.41 | 5.28 | 20.71 | 29.13 | 78.19 | 34.74 | |
| LoRAMRank (r)=16, Backbone=Llama 2-7B, Learning Rate=2 x 10-52026.06 | 40.32 | 5.3 | 18.92 | 28.83 | 75.19 | 33.71 | |
| LoRARank (r)=128, Backbone=Llama 2-7B, Learning Rate=2 x 10-52026.06 | 40.27 | 4.72 | 20.11 | 28.84 | 73.64 | 33.52 | |
| RsLoRARank (r)=16, Backbone=Llama 2-7B, Learning Rate=2 x 10-52026.06 | 39.04 | 4.94 | 18.85 | 28.1 | 73.24 | 32.83 | |
| PiSSARank (r)=16, Backbone=Llama 2-7B, Learning Rate=2 x 10-52026.06 | 37.68 | 5.16 | 18.37 | 28.62 | 73.72 | 32.71 | |
| LoRA+Rank (r)=16, Backbone=Llama 2-7B, Learning Rate=2 x 10-52026.06 | 31.69 | 3.98 | 18.54 | 28 | 72.19 | 30.88 | |
| LoRARank (r)=16, Backbone=Llama 2-7B, Learning Rate=2 x 10-52026.06 | 31.51 | 4.16 | 15.98 | 28.65 | 66.56 | 29.37 |