Single-stream decode on SmolLM2-135M
8.65Latency (ms/token)vLLM cudagraph
Evaluation results
| Method | Links | ||
|---|---|---|---|
| vLLM cudagraphGPU=H100, dtype=float322026.06 | 8.65 | 1.65 | |
| vLLM enforce-eagerGPU=H100, dtype=float322026.06 | 13.51 | 1.06 | |
| AMKGPU=H100, Comparison Baseline=vLLM cudagraph, Optimization=earlier AMK GEMV2026.06 | 14.3 | — | |
| AMKGPU=H100, Comparison Baseline=vLLM enforce-eager, Optimization=earlier AMK GEMV2026.06 | 14.3 | — | |
| AMKGPU=A100, Comparison Baseline=vLLM enforce-eager, Optimization=earlier AMK GEMV2026.06 | 15.59 | 2.06 | |
| vLLM enforce-eagerGPU=A100, dtype=float322026.06 | 32.06 | — |