Language Modeling on LLaMA-2-7B
5.47PerplexityBF16
Evaluation results
| Method | Links | |
|---|---|---|
| BF16Quantization Bits=None, Backbone Model=LLaMA-2-7B2024.02 | 5.47 | |
| FP16Type=Upper bound, Size (GB)=13.52026.03 | 5.51 | |
| RAMPType=Mixed-precision, Size (GB)=3.682026.03 | 5.54 | |
| AWQ-4Type=SOTA, Size (GB)=3.902026.03 | 5.6 | |
| Q4_K_M (GGUF)Type=Deployment, Size (GB)=3.952026.03 | 5.61 | |
| QUIP#-4Type=SOTA, Size (GB)=3.902026.03 | 5.62 | |
| GPTQ-4Type=SOTA, Size (GB)=3.902026.03 | 5.69 | |
| RTN-4Type=Naive, Size (GB)=3.902026.03 | 5.94 | |
| AQLM + PVVariant=2-bit-1x16, Model Size=2.38 GB, Data=8.00M2026.02 | 6.08 | |
| AQLM + PVVariant=2-bit-2x8, Model Size=2.15 GB, Data=8.00M2026.02 | 6.27 | |
| QTIPVariant=2-bit, Model Size=2.15 GB, Data=12.58M2026.02 | 6.29 | |
| AQLMVariant=2-bit-1x16, Model Size=2.38 GB, Data=8.00M2026.02 | 6.34 | |
| AQLMVariant=2-bit-2x8, Model Size=2.15 GB, Data=8.00M2026.02 | 7.24 | |
| NANOQUANTVariant=2-bit, Model Size=1.68 GB, Data=1.05M2026.02 | 7.35 | |
| BitDistillerQuantization Bits=2, Group Size (g)=128, Backbone Model=LLaMA-2-7B2024.02 | 8.08 | |
| QuIP#Quantization Bits=2, Group Size (g)=128, Backbone Model=LLaMA-2-7B2024.02 | 8.97 | |
| NANOQUANTVariant=1-bit, Model Size=1.24 GB, Data=1.05M2026.02 | 9.01 | |
| QuIPQuantization Bits=2, Group Size (g)=128, Backbone Model=LLaMA-2-7B2024.02 | 728.15 |