Text Generation on Decoding Throughput
320Decoding Throughput (tokens/s)AWQ
Evaluation Results
| Method | Links | |
|---|---|---|
| AWQModel=Qwen3-1.7B, Batch size=12025.11 | 320 | |
| PAROQModel=Qwen3-1.7B, Batch size=12025.11 | 278 | |
| QTIPModel=Qwen3-1.7B, Batch size=12025.11 | 209 | |
| AWQModel=Qwen3-4B, Batch size=12025.11 | 176 | |
| FP16Model=Qwen3-1.7B, Batch size=12025.11 | 170 | |
| PAROQModel=Qwen3-4B, Batch size=12025.11 | 160 | |
| AWQModel=LLaMA-3-8B, Batch size=12025.11 | 120 | |
| QTIPModel=Qwen3-4B, Batch size=12025.11 | 117 | |
| PAROQModel=LLaMA-3-8B, Batch size=12025.11 | 112 | |
| QTIPModel=LLaMA-3-8B, Batch size=12025.11 | 95 | |
| FP16Model=Qwen3-4B, Batch size=12025.11 | 78 | |
| AWQModel=Qwen3-14B, Batch size=12025.11 | 70 | |
| PAROQModel=Qwen3-14B, Batch size=12025.11 | 65 | |
| QTIPModel=Qwen3-14B, Batch size=12025.11 | 55 | |
| FP16Model=LLaMA-3-8B, Batch size=12025.11 | 45 | |
| FP16Model=Qwen3-14B, Batch size=12025.11 | 25 |