Decoding on DGX A100 platform
359.3Throughput (tokens/sec)LAQuant
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| LAQuantModel=Qwen3-1.7B, Bits=3, Batch size=12026.05 | 359.3 | 1.73 | |
| LAQuantModel=Llama3.1-8B, Bits=3, Batch size=12026.05 | 294.37 | 3.93 | |
| ParoQuantModel=Qwen3-1.7B, Bits=3, Batch size=12026.05 | 285.4 | 1.37 | |
| LAQuantModel=Qwen3-4B, Bits=3, Batch size=12026.05 | 274.9 | 2.42 | |
| LAQuantModel=Qwen3-1.7B, Bits=4, Batch size=12026.05 | 260.8 | 1.25 | |
| LAQuantModel=Qwen3-8B, Bits=3, Batch size=12026.05 | 256 | 3.52 | |
| ParoQuantModel=Llama3.1-8B, Bits=3, Batch size=12026.05 | 233.73 | 3.12 | |
| ParoQuantModel=Qwen3-1.7B, Bits=4, Batch size=12026.05 | 217.4 | 1.05 | |
| ParoQuantModel=Qwen3-4B, Bits=3, Batch size=12026.05 | 214.9 | 1.89 | |
| FP16Model=Qwen3-1.7B, Bits=16, Batch size=12026.05 | 207.9 | — | |
| ParoQuantModel=Qwen3-8B, Bits=3, Batch size=12026.05 | 203.8 | 2.8 | |
| LAQuantModel=Qwen3-4B, Bits=4, Batch size=12026.05 | 163.1 | 1.43 | |
| LAQuantModel=Llama3.1-8B, Bits=4, Batch size=12026.05 | 143.21 | 1.91 | |
| ParoQuantModel=Qwen3-4B, Bits=4, Batch size=12026.05 | 140.2 | 1.23 | |
| LAQuantModel=Qwen3-8B, Bits=4, Batch size=12026.05 | 131.4 | 1.81 | |
| ParoQuantModel=Llama3.1-8B, Bits=4, Batch size=12026.05 | 126.59 | 1.69 | |
| ParoQuantModel=Qwen3-8B, Bits=4, Batch size=12026.05 | 116.4 | 1.6 | |
| FP16Model=Qwen3-4B, Bits=16, Batch size=12026.05 | 113.8 | — | |
| FP16Model=Llama3.1-8B, Bits=16, Batch size=12026.05 | 74.93 | — | |
| FP16Model=Qwen3-8B, Bits=16, Batch size=12026.05 | 72.8 | — |