Batch-1 Decode Kernel Latency on Llama-shaped Synthetic Models
1.33Speedup Ratio (Median)AMK
Evaluation results
| Method | Links | ||
|---|---|---|---|
| AMKGPU=L4, GPU Architecture (sm)=sm_89, Model Configuration=4B / 3.8 GB, Inference Regime=Datacenter2026.06 | 1.33 | 1.31 | |
| AMKGPU=L4, GPU Architecture (sm)=sm_89, Model Configuration=3.5B / 3.2 GB, Inference Regime=Datacenter2026.06 | 1.32 | 1.29 | |
| AMKGPU=L40S, GPU Architecture (sm)=sm_89, Model Configuration=6.7B / 6.7 GB, Inference Regime=Datacenter2026.06 | 1.27 | 1.25 | |
| AMKGPU=L4, GPU Architecture (sm)=sm_89, Model Configuration=2.7B / 2.5 GB, Inference Regime=Datacenter2026.06 | 1.25 | 1.17 | |
| AMKGPU=L40S, GPU Architecture (sm)=sm_89, Model Configuration=4B / 3.8 GB, Inference Regime=Datacenter2026.06 | 1.25 | 1.22 | |
| AMKGPU=RTX 5090, GPU Architecture (sm)=sm_120, Model Configuration=8L / 984 MB, Inference Regime=Consumer2026.06 | 1.23 | 1.2 | |
| AMKGPU=RTX 5090, GPU Architecture (sm)=sm_120, Model Configuration=4L / 623 MB, Inference Regime=Consumer2026.06 | 1.19 | 1.04 | |
| AMKGPU=L4, GPU Architecture (sm)=sm_89, Model Configuration=1.3B / 1.3 GB, Inference Regime=Datacenter2026.06 | 1.18 | 1.13 | |
| AMKGPU=A10G, GPU Architecture (sm)=sm_86, Model Configuration=4B / 3.8 GB, Inference Regime=Datacenter2026.06 | 1.08 | 1.05 | |
| AMKGPU=A10G, GPU Architecture (sm)=sm_86, Model Configuration=3.5B / 3.2 GB, Inference Regime=Datacenter2026.06 | 1.04 | 1.01 | |
| AMKGPU=A10G, GPU Architecture (sm)=sm_86, Model Configuration=2.7B / 2.5 GB, Inference Regime=Datacenter2026.06 | 1 | 0.97 |