Inference Throughput on Qwen3 Query Projection Module NVIDIA A40
80.63Throughput (k tokens/sec)TTQ (r = 0)
Evaluation Results
| Method | Links | |
|---|---|---|
| TTQ (r = 0)Model Size=0.6B, Hardware=NVIDIA A40 GPU2026.03 | 80.63 | |
| TTQ (r = 0)Model Size=1.7B, Hardware=NVIDIA A40 GPU2026.03 | 78.68 | |
| AWQ (awq_gemm)Model Size=0.6B, Hardware=NVIDIA A40 GPU2026.03 | 75.76 | |
| AWQ (marlin_gemm)Model Size=4B, Hardware=NVIDIA A40 GPU2026.03 | 73.98 | |
| AWQ (marlin_gemm)Model Size=1.7B, Hardware=NVIDIA A40 GPU2026.03 | 73.7 | |
| AWQ (marlin_gemm)Model Size=0.6B, Hardware=NVIDIA A40 GPU2026.03 | 73.32 | |
| AWQ (awq_gemm)Model Size=1.7B, Hardware=NVIDIA A40 GPU2026.03 | 71.6 | |
| TTQ (r = 0)Model Size=4B, Hardware=NVIDIA A40 GPU2026.03 | 69.11 | |
| TTQ (r = 16)Model Size=0.6B, Hardware=NVIDIA A40 GPU2026.03 | 66.67 | |
| TTQ (r = 16)Model Size=1.7B, Hardware=NVIDIA A40 GPU2026.03 | 59.08 | |
| FP16Model Size=0.6B, Hardware=NVIDIA A40 GPU2026.03 | 57.58 | |
| AWQ (awq_gemm)Model Size=4B, Hardware=NVIDIA A40 GPU2026.03 | 54.61 | |
| TTQ (r = 16)Model Size=4B, Hardware=NVIDIA A40 GPU2026.03 | 49.67 | |
| FP16Model Size=1.7B, Hardware=NVIDIA A40 GPU2026.03 | 41.29 | |
| AWQ (marlin_gemm)Model Size=8B, Hardware=NVIDIA A40 GPU2026.03 | 38.91 | |
| AWQ (marlin_gemm)Model Size=14B, Hardware=NVIDIA A40 GPU2026.03 | 37.52 | |
| TTQ (r = 0)Model Size=8B, Hardware=NVIDIA A40 GPU2026.03 | 36.52 | |
| AWQ (awq_gemm)Model Size=8B, Hardware=NVIDIA A40 GPU2026.03 | 32.68 | |
| AWQ (marlin_gemm)Model Size=32B, Hardware=NVIDIA A40 GPU2026.03 | 30.32 | |
| TTQ (r = 16)Model Size=8B, Hardware=NVIDIA A40 GPU2026.03 | 30.32 | |
| AWQ (awq_gemm)Model Size=14B, Hardware=NVIDIA A40 GPU2026.03 | 25.28 | |
| TTQ (r = 0)Model Size=14B, Hardware=NVIDIA A40 GPU2026.03 | 24.94 | |
| TTQ (r = 16)Model Size=14B, Hardware=NVIDIA A40 GPU2026.03 | 23.78 | |
| FP16Model Size=4B, Hardware=NVIDIA A40 GPU2026.03 | 21.38 | |
| TTQ (r = 0)Model Size=32B, Hardware=NVIDIA A40 GPU2026.03 | 19.34 | |
| AWQ (awq_gemm)Model Size=32B, Hardware=NVIDIA A40 GPU2026.03 | 18.28 | |
| TTQ (r = 16)Model Size=32B, Hardware=NVIDIA A40 GPU2026.03 | 17.43 | |
| FP16Model Size=8B, Hardware=NVIDIA A40 GPU2026.03 | 14.1 | |
| FP16Model Size=14B, Hardware=NVIDIA A40 GPU2026.03 | 10.43 | |
| FP16Model Size=32B, Hardware=NVIDIA A40 GPU2026.03 | 6.86 |