List your company
ResearchFormerly WizWand

Inference Throughput on Qwen3 Models (test)

120.62Throughput (k tokens/sec)AWQ
6.365636.027865.6995.3522Mar 11, 2026
Updated 7mo ago

Evaluation results

MethodLinks
AWQModel Size=0.6B, Quantization=4-bit, Kernel Implementation=marlin_gemm, GPU=NVIDIA RTX40902026.03120.62
FP16Model Size=0.6B, Precision=FP16, GPU=NVIDIA RTX40902026.03116.82
AWQModel Size=1.7B, Quantization=4-bit, Kernel Implementation=marlin_gemm, GPU=NVIDIA RTX40902026.03115.2
AWQModel Size=0.6B, Quantization=4-bit, Kernel Implementation=awq_gemm, GPU=NVIDIA RTX40902026.03113.84
AWQModel Size=4B, Quantization=4-bit, Kernel Implementation=marlin_gemm, GPU=NVIDIA RTX40902026.03112.61
TTQModel Size=0.6B, Quantization=4-bit, Rank (r)=0, GPU=NVIDIA RTX40902026.03108
TTQModel Size=1.7B, Quantization=4-bit, Rank (r)=0, GPU=NVIDIA RTX40902026.03104.11
TTQModel Size=4B, Quantization=4-bit, Rank (r)=0, GPU=NVIDIA RTX40902026.03103.45
AWQModel Size=8B, Quantization=4-bit, Kernel Implementation=marlin_gemm, GPU=NVIDIA RTX40902026.03101
TTQModel Size=8B, Quantization=4-bit, Rank (r)=0, GPU=NVIDIA RTX40902026.0392.23
AWQModel Size=1.7B, Quantization=4-bit, Kernel Implementation=awq_gemm, GPU=NVIDIA RTX40902026.0390.42
AWQModel Size=14B, Quantization=4-bit, Kernel Implementation=marlin_gemm, GPU=NVIDIA RTX40902026.0380.45
AWQModel Size=4B, Quantization=4-bit, Kernel Implementation=awq_gemm, GPU=NVIDIA RTX40902026.0380.19
TTQModel Size=0.6B, Quantization=4-bit, Rank (r)=16, GPU=NVIDIA RTX40902026.0377.88
FP16Model Size=1.7B, Precision=FP16, GPU=NVIDIA RTX40902026.0377.04
TTQModel Size=14B, Quantization=4-bit, Rank (r)=0, GPU=NVIDIA RTX40902026.0374.72
TTQModel Size=1.7B, Quantization=4-bit, Rank (r)=16, GPU=NVIDIA RTX40902026.0373.47
TTQModel Size=4B, Quantization=4-bit, Rank (r)=16, GPU=NVIDIA RTX40902026.0372.55
FP16Model Size=4B, Precision=FP16, GPU=NVIDIA RTX40902026.0372.45
AWQModel Size=32B, Quantization=4-bit, Kernel Implementation=marlin_gemm, GPU=NVIDIA RTX40902026.0372.34
TTQModel Size=32B, Quantization=4-bit, Rank (r)=0, GPU=NVIDIA RTX40902026.0367.4
TTQModel Size=8B, Quantization=4-bit, Rank (r)=16, GPU=NVIDIA RTX40902026.0366.62
AWQModel Size=8B, Quantization=4-bit, Kernel Implementation=awq_gemm, GPU=NVIDIA RTX40902026.0362.94
TTQModel Size=14B, Quantization=4-bit, Rank (r)=16, GPU=NVIDIA RTX40902026.0358.65
FP16Model Size=8B, Precision=FP16, GPU=NVIDIA RTX40902026.0358.44
TTQModel Size=32B, Quantization=4-bit, Rank (r)=16, GPU=NVIDIA RTX40902026.0353.17
AWQModel Size=14B, Quantization=4-bit, Kernel Implementation=awq_gemm, GPU=NVIDIA RTX40902026.0351.32
AWQModel Size=32B, Quantization=4-bit, Kernel Implementation=awq_gemm, GPU=NVIDIA RTX40902026.0350.45
FP16Model Size=14B, Precision=FP16, GPU=NVIDIA RTX40902026.0346.78
FP16Model Size=32B, Precision=FP16, GPU=NVIDIA RTX40902026.0310.76