INT2 Quantization on LLaMA 7B
8.76Memory Cost (GB)OPTQ
Evaluation results
| Method | Links | |
|---|---|---|
| OPTQTarget=layer-wise reconstruction, Hardware=NVIDIA A100 GPU2024.02 | 8.76 | |
| Z-FOLDTarget=layer-wise reconstruction, Hardware=NVIDIA A100 GPU2024.02 | 8.76 | |
| OPTQTarget=layer-wise reconstruction, Hardware=single NVIDIA A100 GPU2024.02 | 8.76 | |
| Z-FOLDTarget=layer-wise reconstruction, Hardware=single NVIDIA A100 GPU2024.02 | 8.76 | |
| OmniQuantTarget=attention-wise reconstruction, Hardware=NVIDIA A100 GPU2024.02 | 12.61 | |
| OmniQuantTarget=attention-wise reconstruction, Hardware=single NVIDIA A100 GPU2024.02 | 12.61 | |
| aespaTarget=attention-wise reconstruction, Hardware=NVIDIA A100 GPU2024.02 | 21.69 | |
| aespaTarget=attention-wise reconstruction, Hardware=single NVIDIA A100 GPU2024.02 | 21.69 | |
| AffineQuantTarget=attention-wise reconstruction, Hardware=NVIDIA A100 GPU2024.02 | 24.28 | |
| AffineQuantTarget=attention-wise reconstruction, Hardware=single NVIDIA A100 GPU2024.02 | 24.28 |