Language Modeling on WikiText-2 vLLM harness (test)
8.87Perplexity (PPL)Llama 3.1-8B-Instruct: Baseline (FP16)
Evaluation Results
| Method | Links | |
|---|---|---|
| Llama 3.1-8B-Instruct: Baseline (FP16)Method=Full precision, Disk (GB)=15.0, GPU (GB)=14.962026.03 | 8.87 | |
| GGUF Q5 K SMethod=Uniform quant., Disk (GB)=5.3, GPU (GB)=5.22026.03 | 8.99 | |
| FP8Method=Uniform quant., Disk (GB)=9.1, GPU (GB)=8.462026.03 | 9.04 | |
| GPTQ INT4Method=Uniform quant., Disk (GB)=5.7, GPU (GB)=5.32026.03 | 9.3 | |
| Clustered (Orig) — no trainingMethod=K-means, Disk (GB)=6.9, GPU (GB)=14.962026.03 | 9.32 | |
| AWQ INT4Method=Uniform quant., Disk (GB)=5.7, GPU (GB)=5.32026.03 | 9.35 | |
| AQLM 2-bitMethod=Codebook, Disk (GB)=4.1, GPU (GB)=3.802026.03 | 11.77 | |
| Compressed 3B-LlamaMethod=CompactifAI, Disk (GB)=6.1, GPU (GB)=6.012026.03 | 12.62 | |
| Compressed + Clustered + Fine-tunedMethod=CompactifAI + K-means, Disk (GB)=3.2, GPU (GB)=6.012026.03 | 13.05 | |
| Compressed + ClusteredMethod=CompactifAI + K-means, Disk (GB)=3.2, GPU (GB)=6.012026.03 | 13.36 | |
| Compressed + Clustered + AWQMethod=CompactifAI + K-means + INT4, Disk (GB)=2.6, GPU (GB)=2.552026.03 | 13.86 | |
| Compressed + Clustered + GPTQMethod=CompactifAI + K-means + INT4, Disk (GB)=2.6, GPU (GB)=2.562026.03 | 14.21 |