Decoding Throughput on Llama 3.1 8B
225.01Throughput (TOK/s)LAQuant
Evaluation results
| Method | Links | ||
|---|---|---|---|
| LAQuantBits=3, Batch size=1, Hardware=RTX A60002026.05 | 225.01 | 5.56 | |
| ParoQuantBits=3, Batch size=1, Hardware=RTX A60002026.05 | 198.13 | 4.9 | |
| LAQuantBits=4, Batch size=1, Hardware=RTX A60002026.05 | 99.74 | 2.46 | |
| ParoQuantBits=4, Batch size=1, Hardware=RTX A60002026.05 | 94.35 | 2.33 | |
| FP16Bits=16, Batch size=1, Hardware=RTX A60002026.05 | 40.47 | — |