Inference Throughput on Qwen3 Models (test)
120.62Throughput (k tokens/sec)AWQ
Evaluation Results
| Method | Links | |
|---|---|---|
| AWQModel Size=0.6B, Quantization=4-bit, Kernel Implementation=marlin_gemm, GPU=NVIDIA RTX40902026.03 | 120.62 | |
| FP16Model Size=0.6B, Precision=FP16, GPU=NVIDIA RTX40902026.03 | 116.82 | |
| AWQModel Size=1.7B, Quantization=4-bit, Kernel Implementation=marlin_gemm, GPU=NVIDIA RTX40902026.03 | 115.2 | |
| AWQModel Size=0.6B, Quantization=4-bit, Kernel Implementation=awq_gemm, GPU=NVIDIA RTX40902026.03 | 113.84 | |
| AWQModel Size=4B, Quantization=4-bit, Kernel Implementation=marlin_gemm, GPU=NVIDIA RTX40902026.03 | 112.61 | |
| TTQModel Size=0.6B, Quantization=4-bit, Rank (r)=0, GPU=NVIDIA RTX40902026.03 | 108 | |
| TTQModel Size=1.7B, Quantization=4-bit, Rank (r)=0, GPU=NVIDIA RTX40902026.03 | 104.11 | |
| TTQModel Size=4B, Quantization=4-bit, Rank (r)=0, GPU=NVIDIA RTX40902026.03 | 103.45 | |
| AWQModel Size=8B, Quantization=4-bit, Kernel Implementation=marlin_gemm, GPU=NVIDIA RTX40902026.03 | 101 | |
| TTQModel Size=8B, Quantization=4-bit, Rank (r)=0, GPU=NVIDIA RTX40902026.03 | 92.23 | |
| AWQModel Size=1.7B, Quantization=4-bit, Kernel Implementation=awq_gemm, GPU=NVIDIA RTX40902026.03 | 90.42 | |
| AWQModel Size=14B, Quantization=4-bit, Kernel Implementation=marlin_gemm, GPU=NVIDIA RTX40902026.03 | 80.45 | |
| AWQModel Size=4B, Quantization=4-bit, Kernel Implementation=awq_gemm, GPU=NVIDIA RTX40902026.03 | 80.19 | |
| TTQModel Size=0.6B, Quantization=4-bit, Rank (r)=16, GPU=NVIDIA RTX40902026.03 | 77.88 | |
| FP16Model Size=1.7B, Precision=FP16, GPU=NVIDIA RTX40902026.03 | 77.04 | |
| TTQModel Size=14B, Quantization=4-bit, Rank (r)=0, GPU=NVIDIA RTX40902026.03 | 74.72 | |
| TTQModel Size=1.7B, Quantization=4-bit, Rank (r)=16, GPU=NVIDIA RTX40902026.03 | 73.47 | |
| TTQModel Size=4B, Quantization=4-bit, Rank (r)=16, GPU=NVIDIA RTX40902026.03 | 72.55 | |
| FP16Model Size=4B, Precision=FP16, GPU=NVIDIA RTX40902026.03 | 72.45 | |
| AWQModel Size=32B, Quantization=4-bit, Kernel Implementation=marlin_gemm, GPU=NVIDIA RTX40902026.03 | 72.34 | |
| TTQModel Size=32B, Quantization=4-bit, Rank (r)=0, GPU=NVIDIA RTX40902026.03 | 67.4 | |
| TTQModel Size=8B, Quantization=4-bit, Rank (r)=16, GPU=NVIDIA RTX40902026.03 | 66.62 | |
| AWQModel Size=8B, Quantization=4-bit, Kernel Implementation=awq_gemm, GPU=NVIDIA RTX40902026.03 | 62.94 | |
| TTQModel Size=14B, Quantization=4-bit, Rank (r)=16, GPU=NVIDIA RTX40902026.03 | 58.65 | |
| FP16Model Size=8B, Precision=FP16, GPU=NVIDIA RTX40902026.03 | 58.44 | |
| TTQModel Size=32B, Quantization=4-bit, Rank (r)=16, GPU=NVIDIA RTX40902026.03 | 53.17 | |
| AWQModel Size=14B, Quantization=4-bit, Kernel Implementation=awq_gemm, GPU=NVIDIA RTX40902026.03 | 51.32 | |
| AWQModel Size=32B, Quantization=4-bit, Kernel Implementation=awq_gemm, GPU=NVIDIA RTX40902026.03 | 50.45 | |
| FP16Model Size=14B, Precision=FP16, GPU=NVIDIA RTX40902026.03 | 46.78 | |
| FP16Model Size=32B, Precision=FP16, GPU=NVIDIA RTX40902026.03 | 10.76 |