List your company
ResearchFormerly WizWand

Inference Throughput on Qwen3 Query Projection Module NVIDIA A40

80.63Throughput (k tokens/sec)TTQ (r = 0)
3.909223.827143.74563.6629Mar 11, 2026
Updated 7mo ago

Evaluation results

MethodLinks
TTQ (r = 0)Model Size=0.6B, Hardware=NVIDIA A40 GPU2026.0380.63
TTQ (r = 0)Model Size=1.7B, Hardware=NVIDIA A40 GPU2026.0378.68
AWQ (awq_gemm)Model Size=0.6B, Hardware=NVIDIA A40 GPU2026.0375.76
AWQ (marlin_gemm)Model Size=4B, Hardware=NVIDIA A40 GPU2026.0373.98
AWQ (marlin_gemm)Model Size=1.7B, Hardware=NVIDIA A40 GPU2026.0373.7
AWQ (marlin_gemm)Model Size=0.6B, Hardware=NVIDIA A40 GPU2026.0373.32
AWQ (awq_gemm)Model Size=1.7B, Hardware=NVIDIA A40 GPU2026.0371.6
TTQ (r = 0)Model Size=4B, Hardware=NVIDIA A40 GPU2026.0369.11
TTQ (r = 16)Model Size=0.6B, Hardware=NVIDIA A40 GPU2026.0366.67
TTQ (r = 16)Model Size=1.7B, Hardware=NVIDIA A40 GPU2026.0359.08
FP16Model Size=0.6B, Hardware=NVIDIA A40 GPU2026.0357.58
AWQ (awq_gemm)Model Size=4B, Hardware=NVIDIA A40 GPU2026.0354.61
TTQ (r = 16)Model Size=4B, Hardware=NVIDIA A40 GPU2026.0349.67
FP16Model Size=1.7B, Hardware=NVIDIA A40 GPU2026.0341.29
AWQ (marlin_gemm)Model Size=8B, Hardware=NVIDIA A40 GPU2026.0338.91
AWQ (marlin_gemm)Model Size=14B, Hardware=NVIDIA A40 GPU2026.0337.52
TTQ (r = 0)Model Size=8B, Hardware=NVIDIA A40 GPU2026.0336.52
AWQ (awq_gemm)Model Size=8B, Hardware=NVIDIA A40 GPU2026.0332.68
AWQ (marlin_gemm)Model Size=32B, Hardware=NVIDIA A40 GPU2026.0330.32
TTQ (r = 16)Model Size=8B, Hardware=NVIDIA A40 GPU2026.0330.32
AWQ (awq_gemm)Model Size=14B, Hardware=NVIDIA A40 GPU2026.0325.28
TTQ (r = 0)Model Size=14B, Hardware=NVIDIA A40 GPU2026.0324.94
TTQ (r = 16)Model Size=14B, Hardware=NVIDIA A40 GPU2026.0323.78
FP16Model Size=4B, Hardware=NVIDIA A40 GPU2026.0321.38
TTQ (r = 0)Model Size=32B, Hardware=NVIDIA A40 GPU2026.0319.34
AWQ (awq_gemm)Model Size=32B, Hardware=NVIDIA A40 GPU2026.0318.28
TTQ (r = 16)Model Size=32B, Hardware=NVIDIA A40 GPU2026.0317.43
FP16Model Size=8B, Hardware=NVIDIA A40 GPU2026.0314.1
FP16Model Size=14B, Hardware=NVIDIA A40 GPU2026.0310.43
FP16Model Size=32B, Hardware=NVIDIA A40 GPU2026.036.86