LLM Inference on Qwen2.5-3B-Instruct K3V3
17.84Median Decode Latency (ms/step)FP16 FA-2
Evaluation Results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| FP16 FA-2Context Length (T)=16K, Hardware (GPU)=H8002026.06 | 17.84 | 17.86 | 17.76 | 18.01 | 0.37 | 1 | |
| FP16 FA-2Context Length (T)=64K, Hardware (GPU)=H8002026.06 | 36.15 | 36.15 | 36.12 | 36.17 | 2.84 | 1 | |
| TQ-MSEContext Length (T)=16K, Hardware (GPU)=H8002026.06 | 45.47 | 45.46 | 45.32 | 45.61 | 0.87 | 0.39 | |
| TQ-MSEContext Length (T)=128K, Hardware (GPU)=H8002026.06 | 45.53 | 45.63 | 45.39 | 45.97 | 36.41 | 1.56 | |
| Block-GTQContext Length (T)=64K, Hardware (GPU)=H8002026.06 | 45.57 | 45.56 | 45.44 | 45.63 | 16.62 | 0.79 | |
| TQ-MSEContext Length (T)=64K, Hardware (GPU)=H8002026.06 | 45.58 | 45.59 | 45.41 | 45.68 | 9.88 | 0.79 | |
| Block-GTQContext Length (T)=16K, Hardware (GPU)=H8002026.06 | 45.86 | 45.83 | 45.66 | 46.01 | 1.33 | 0.39 | |
| Block-GTQContext Length (T)=128K, Hardware (GPU)=H8002026.06 | 52.95 | 60.52 | 52.69 | 121.72 | 63.53 | 1.34 | |
| TQ-MSEContext Length (T)=256K, Hardware (GPU)=H8002026.06 | 60.95 | 60.97 | 60.76 | 61.21 | 141.5 | — | |
| FP16 FA-2Context Length (T)=128K, Hardware (GPU)=H8002026.06 | 70.96 | 70.95 | 70.85 | 71.03 | 9.09 | 1 | |
| Block-GTQContext Length (T)=256K, Hardware (GPU)=H8002026.06 | 82.24 | 82.27 | 81.97 | 82.61 | 250.17 | — | |
| TQ-MSEContext Length (T)=512K, Hardware (GPU)=H8002026.06 | 101.18 | 101.22 | 101.02 | 101.45 | 558.75 | — | |
| Block-GTQContext Length (T)=512K, Hardware (GPU)=H8002026.06 | 140.27 | 140.4 | 140.04 | 140.86 | 997.19 | — |