Runtime Speed on Qwen3 Query Projection Module
92.57Throughput (k tokens/sec)TTQ
Evaluation results
| Method | Links | |
|---|---|---|
| TTQModel Scale=1.7B, Hardware=NVIDIA L40 GPU, Quantization=4-bit, Rank (r)=02026.03 | 92.57 | |
| TTQModel Scale=4B, Hardware=NVIDIA L40 GPU, Quantization=4-bit, Rank (r)=02026.03 | 91.86 | |
| FP16Model Scale=0.6B, Hardware=NVIDIA L40 GPU, Precision=FP162026.03 | 91.35 | |
| AWQ (marlin_gemm)Model Scale=1.7B, Hardware=NVIDIA L40 GPU, Quantization=4-bit, Kernel Implementation=marlin_gemm2026.03 | 91.35 | |
| AWQ (marlin_gemm)Model Scale=4B, Hardware=NVIDIA L40 GPU, Quantization=4-bit, Kernel Implementation=marlin_gemm2026.03 | 91.28 | |
| AWQ (awq_gemm)Model Scale=0.6B, Hardware=NVIDIA L40 GPU, Quantization=4-bit, Kernel Implementation=awq_gemm2026.03 | 91.01 | |
| AWQ (marlin_gemm)Model Scale=8B, Hardware=NVIDIA L40 GPU, Quantization=4-bit, Kernel Implementation=marlin_gemm2026.03 | 90.93 | |
| TTQModel Scale=0.6B, Hardware=NVIDIA L40 GPU, Quantization=4-bit, Rank (r)=02026.03 | 90.59 | |
| AWQ (marlin_gemm)Model Scale=0.6B, Hardware=NVIDIA L40 GPU, Quantization=4-bit, Kernel Implementation=marlin_gemm2026.03 | 86.57 | |
| AWQ (awq_gemm)Model Scale=1.7B, Hardware=NVIDIA L40 GPU, Quantization=4-bit, Kernel Implementation=awq_gemm2026.03 | 84.7 | |
| TTQModel Scale=8B, Hardware=NVIDIA L40 GPU, Quantization=4-bit, Rank (r)=02026.03 | 84.19 | |
| AWQ (marlin_gemm)Model Scale=14B, Hardware=NVIDIA L40 GPU, Quantization=4-bit, Kernel Implementation=marlin_gemm2026.03 | 77.81 | |
| AWQ (awq_gemm)Model Scale=4B, Hardware=NVIDIA L40 GPU, Quantization=4-bit, Kernel Implementation=awq_gemm2026.03 | 76 | |
| FP16Model Scale=1.7B, Hardware=NVIDIA L40 GPU, Precision=FP162026.03 | 75.18 | |
| FP16Model Scale=4B, Hardware=NVIDIA L40 GPU, Precision=FP162026.03 | 74.6 | |
| TTQModel Scale=0.6B, Hardware=NVIDIA L40 GPU, Quantization=4-bit, Rank (r)=162026.03 | 72.82 | |
| TTQModel Scale=14B, Hardware=NVIDIA L40 GPU, Quantization=4-bit, Rank (r)=02026.03 | 71.28 | |
| TTQModel Scale=4B, Hardware=NVIDIA L40 GPU, Quantization=4-bit, Rank (r)=162026.03 | 69.84 | |
| AWQModel Size=0.6B, Kernel Implementation=marlin_gemm, Hardware=NVIDIA A100 GPU2026.03 | 69.41 | |
| TTQModel Scale=1.7B, Hardware=NVIDIA L40 GPU, Quantization=4-bit, Rank (r)=162026.03 | 69.11 | |
| FP16Model Size=1.7B, Hardware=NVIDIA A100 GPU2026.03 | 68.91 | |
| FP16Model Size=0.6B, Hardware=NVIDIA A100 GPU2026.03 | 68.43 | |
| AWQ (marlin_gemm)Model Scale=32B, Hardware=NVIDIA L40 GPU, Quantization=4-bit, Kernel Implementation=marlin_gemm2026.03 | 68.15 | |
| AWQModel Size=0.6B, Kernel Implementation=awq_gemm, Hardware=NVIDIA A100 GPU2026.03 | 68.1 | |
| TTQModel Size=0.6B, Rank (r)=0, Quantization=4-bit, Hardware=NVIDIA RTX3090 GPU2026.03 | 68.07 | |
| AWQModel Size=1.7B, Kernel Implementation=marlin_gemm, Quantization=4-bit, Hardware=NVIDIA RTX3090 GPU2026.03 | 67.86 | |
| TTQModel Size=1.7B, Rank (r)=0, Quantization=4-bit, Hardware=NVIDIA RTX3090 GPU2026.03 | 67.34 | |
| AWQModel Size=4B, Kernel Implementation=marlin_gemm, Quantization=4-bit, Hardware=NVIDIA RTX3090 GPU2026.03 | 65.88 | |
| AWQModel Size=1.7B, Kernel Implementation=marlin_gemm, Hardware=NVIDIA A100 GPU2026.03 | 64.93 | |
| TTQModel Size=4B, Rank (r)=0, Quantization=4-bit, Hardware=NVIDIA RTX3090 GPU2026.03 | 64.43 | |
| TTQModel Scale=32B, Hardware=NVIDIA L40 GPU, Quantization=4-bit, Rank (r)=02026.03 | 63.37 | |
| TTQModel Scale=8B, Hardware=NVIDIA L40 GPU, Quantization=4-bit, Rank (r)=162026.03 | 63.37 | |
| AWQModel Size=0.6B, Kernel Implementation=marlin_gemm, Quantization=4-bit, Hardware=NVIDIA RTX3090 GPU2026.03 | 63.13 | |
| AWQModel Size=0.6B, Kernel Implementation=awq_gemm, Quantization=4-bit, Hardware=NVIDIA RTX3090 GPU2026.03 | 62.89 | |
| TTQModel Size=0.6B, TTQ Rank (r)=0, Hardware=NVIDIA A100 GPU2026.03 | 61.85 | |
| AWQModel Size=1.7B, Kernel Implementation=awq_gemm, Quantization=4-bit, Hardware=NVIDIA RTX3090 GPU2026.03 | 60.54 | |
| AWQ (awq_gemm)Model Scale=8B, Hardware=NVIDIA L40 GPU, Quantization=4-bit, Kernel Implementation=awq_gemm2026.03 | 59.99 | |
| FP16Model Size=0.6B, Hardware=NVIDIA RTX3090 GPU2026.03 | 59.75 | |
| TTQModel Size=1.7B, Rank (r)=16, Quantization=4-bit, Hardware=NVIDIA RTX3090 GPU2026.03 | 58.72 | |
| AWQModel Size=4B, Kernel Implementation=marlin_gemm, Hardware=NVIDIA A100 GPU2026.03 | 58.41 | |
| TTQModel Size=1.7B, TTQ Rank (r)=0, Hardware=NVIDIA A100 GPU2026.03 | 57.72 | |
| TTQModel Size=0.6B, Rank (r)=16, Quantization=4-bit, Hardware=NVIDIA RTX3090 GPU2026.03 | 57.57 | |
| AWQModel Size=1.7B, Kernel Implementation=awq_gemm, Hardware=NVIDIA A100 GPU2026.03 | 56.09 | |
| TTQModel Scale=14B, Hardware=NVIDIA L40 GPU, Quantization=4-bit, Rank (r)=162026.03 | 55.87 | |
| FP16Model Size=1.7B, Hardware=NVIDIA RTX3090 GPU2026.03 | 54.38 | |
| FP16Model Scale=8B, Hardware=NVIDIA L40 GPU, Precision=FP162026.03 | 53.31 | |
| AWQModel Size=8B, Kernel Implementation=marlin_gemm, Hardware=NVIDIA A100 GPU2026.03 | 53.13 | |
| TTQModel Size=4B, TTQ Rank (r)=0, Hardware=NVIDIA A100 GPU2026.03 | 52.67 | |
| TTQModel Scale=32B, Hardware=NVIDIA L40 GPU, Quantization=4-bit, Rank (r)=162026.03 | 49.62 | |
| AWQModel Size=14B, Kernel Implementation=marlin_gemm, Hardware=NVIDIA A100 GPU2026.03 | 49.07 | |
| AWQ (awq_gemm)Model Scale=14B, Hardware=NVIDIA L40 GPU, Quantization=4-bit, Kernel Implementation=awq_gemm2026.03 | 48.95 | |
| TTQModel Size=8B, TTQ Rank (r)=0, Hardware=NVIDIA A100 GPU2026.03 | 48.58 | |
| AWQModel Size=4B, Kernel Implementation=awq_gemm, Quantization=4-bit, Hardware=NVIDIA RTX3090 GPU2026.03 | 48.39 | |
| AWQModel Size=4B, Kernel Implementation=awq_gemm, Hardware=NVIDIA A100 GPU2026.03 | 48.2 | |
| AWQ (awq_gemm)Model Scale=32B, Hardware=NVIDIA L40 GPU, Quantization=4-bit, Kernel Implementation=awq_gemm2026.03 | 47.87 | |
| AWQModel Size=8B, Kernel Implementation=marlin_gemm, Quantization=4-bit, Hardware=NVIDIA RTX3090 GPU2026.03 | 46.97 | |
| TTQModel Size=0.6B, TTQ Rank (r)=16, Hardware=NVIDIA A100 GPU2026.03 | 46.04 | |
| TTQModel Size=14B, TTQ Rank (r)=0, Hardware=NVIDIA A100 GPU2026.03 | 45.17 | |
| TTQModel Size=4B, Rank (r)=16, Quantization=4-bit, Hardware=NVIDIA RTX3090 GPU2026.03 | 44.73 | |
| FP16Model Scale=14B, Hardware=NVIDIA L40 GPU, Precision=FP162026.03 | 44.45 | |
| TTQModel Size=1.7B, TTQ Rank (r)=16, Hardware=NVIDIA A100 GPU2026.03 | 43.11 | |
| TTQModel Size=8B, Rank (r)=0, Quantization=4-bit, Hardware=NVIDIA RTX3090 GPU2026.03 | 43.1 | |
| FP16Model Size=4B, Hardware=NVIDIA A100 GPU2026.03 | 42.76 | |
| AWQModel Size=32B, Kernel Implementation=marlin_gemm, Hardware=NVIDIA A100 GPU2026.03 | 42.28 | |
| TTQModel Size=4B, TTQ Rank (r)=16, Hardware=NVIDIA A100 GPU2026.03 | 40.01 | |
| TTQModel Size=32B, TTQ Rank (r)=0, Hardware=NVIDIA A100 GPU2026.03 | 39.4 | |
| AWQModel Size=8B, Kernel Implementation=awq_gemm, Hardware=NVIDIA A100 GPU2026.03 | 38.28 | |
| TTQModel Size=8B, TTQ Rank (r)=16, Hardware=NVIDIA A100 GPU2026.03 | 37.19 | |
| AWQModel Size=14B, Kernel Implementation=marlin_gemm, Quantization=4-bit, Hardware=NVIDIA RTX3090 GPU2026.03 | 36.3 | |
| FP16Model Scale=32B, Hardware=NVIDIA L40 GPU, Precision=FP162026.03 | 34.74 | |
| TTQModel Size=14B, TTQ Rank (r)=16, Hardware=NVIDIA A100 GPU2026.03 | 34.32 | |
| TTQModel Size=14B, Rank (r)=0, Quantization=4-bit, Hardware=NVIDIA RTX3090 GPU2026.03 | 34.25 | |
| AWQModel Size=8B, Kernel Implementation=awq_gemm, Quantization=4-bit, Hardware=NVIDIA RTX3090 GPU2026.03 | 33.48 | |
| TTQModel Size=8B, Rank (r)=16, Quantization=4-bit, Hardware=NVIDIA RTX3090 GPU2026.03 | 33.2 | |
| AWQModel Size=14B, Kernel Implementation=awq_gemm, Hardware=NVIDIA A100 GPU2026.03 | 32.25 | |
| FP16Model Size=8B, Hardware=NVIDIA A100 GPU2026.03 | 30.69 | |
| TTQModel Size=32B, TTQ Rank (r)=16, Hardware=NVIDIA A100 GPU2026.03 | 30.46 | |
| FP16Model Size=4B, Hardware=NVIDIA RTX3090 GPU2026.03 | 29.53 | |
| TTQModel Size=14B, Rank (r)=16, Quantization=4-bit, Hardware=NVIDIA RTX3090 GPU2026.03 | 28.45 | |
| AWQModel Size=14B, Kernel Implementation=awq_gemm, Quantization=4-bit, Hardware=NVIDIA RTX3090 GPU2026.03 | 27.92 | |
| AWQModel Size=32B, Kernel Implementation=marlin_gemm, Quantization=4-bit, Hardware=NVIDIA RTX3090 GPU2026.03 | 26.17 | |
| TTQModel Size=32B, Rank (r)=0, Quantization=4-bit, Hardware=NVIDIA RTX3090 GPU2026.03 | 24.96 | |
| AWQModel Size=32B, Kernel Implementation=awq_gemm, Hardware=NVIDIA A100 GPU2026.03 | 24.35 | |
| TTQModel Size=32B, Rank (r)=16, Quantization=4-bit, Hardware=NVIDIA RTX3090 GPU2026.03 | 21.85 | |
| AWQModel Size=32B, Kernel Implementation=awq_gemm, Quantization=4-bit, Hardware=NVIDIA RTX3090 GPU2026.03 | 21.62 | |
| FP16Model Size=8B, Hardware=NVIDIA RTX3090 GPU2026.03 | 20.37 | |
| FP16Model Size=14B, Hardware=NVIDIA A100 GPU2026.03 | 20.16 | |
| FP16Model Size=32B, Hardware=NVIDIA A100 GPU2026.03 | 15.9 | |
| FP16Model Size=14B, Hardware=NVIDIA RTX3090 GPU2026.03 | 14.15 | |
| FP16Model Size=32B, Hardware=NVIDIA RTX3090 GPU2026.03 | 9.77 |