Fine-tuning Resource Usage on LLaMA-2-7B
7.42Weights Memory UsageQFT
Evaluation Results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| QFT2023.10 | 7.42 | 7.06 | — | 7.06 | — | 3.75 | 25.3 | 28.9 | |
| Adam-FP16 mixedPrecision=FP16 mixed2023.10 | 12.6 | 12.6 | 25.1 | 25.1 | 25.1 | 3.75 | 104 | 123 | |
| Adam-FP32Precision=FP322023.10 | 25.1 | 25.1 | — | 25.1 | 25.1 | 3.75 | 104 | 129 | |
| Bitsandbytes2023.10 | 25.1 | 25.1 | — | 6.31 | 6.31 | 3.75 | 66.6 | 86.6 | |
| Lion-FP32Precision=FP322023.10 | 25.1 | 25.1 | — | 25.1 | — | 3.75 | 79.1 | 101 |