Large Language Model Inference on Llama-3-8B
130.7Throughput (Tok/s)GPTQ
Evaluation results
| Method | Links | ||
|---|---|---|---|
| GPTQHardware=Single NVIDIA A100 GPU, Batch size=1, Generation length=1024 tokens2026.06 | 130.7 | 4.14 | |
| UniSVQHardware=Single NVIDIA A100 GPU, Batch size=1, Generation length=1024 tokens2026.06 | 101.65 | 3.87 | |
| QUIP#Hardware=Single NVIDIA A100 GPU, Batch size=1, Generation length=1024 tokens2026.06 | 79.6 | 4.13 | |
| AQLMHardware=Single NVIDIA A100 GPU, Batch size=1, Generation length=1024 tokens, Group size=2x82026.06 | 70.97 | 4.44 | |
| FP16Hardware=Single NVIDIA A100 GPU, Batch size=1, Generation length=1024 tokens2026.06 | 60.38 | 15.72 |