LLM Decoding on Llama 70B (H100 GPU Cluster) 3.1
894.32ThroughputDeepFusionKernel
Evaluation Results
| Method | Links | |
|---|---|---|
| DeepFusionKernelBatch size=16, Output len=1024, Kernel scheduler=true2026.02 | 894.32 | |
| SGLangBatch size=16, Output len=1024, Kernel scheduler=true2026.02 | 878.08 | |
| vLLMBatch size=16, Output len=1024, Kernel scheduler=true2026.02 | 865.37 | |
| DeepFusionKernelBatch size=16, Output len=4096, Kernel scheduler=true2026.02 | 863.4 | |
| vLLMBatch size=16, Output len=4096, Kernel scheduler=true2026.02 | 850.33 | |
| SGLangBatch size=16, Output len=4096, Kernel scheduler=true2026.02 | 815.4 | |
| DeepFusionKernelBatch size=16, Output len=16384, Kernel scheduler=true2026.02 | 764.57 | |
| vLLMBatch size=16, Output len=16384, Kernel scheduler=true2026.02 | 746.42 | |
| SGLangBatch size=16, Output len=16384, Kernel scheduler=true2026.02 | 732.18 | |
| DeepFusionKernelBatch size=4, Output len=4096, Kernel scheduler=true2026.02 | 234.61 | |
| DeepFusionKernelBatch size=4, Output len=1024, Kernel scheduler=true2026.02 | 230.98 | |
| SGLangBatch size=4, Output len=1024, Kernel scheduler=true2026.02 | 228.9 | |
| DeepFusionKernelBatch size=4, Output len=16384, Kernel scheduler=true2026.02 | 227.73 | |
| vLLMBatch size=4, Output len=4096, Kernel scheduler=true2026.02 | 227.7 | |
| vLLMBatch size=4, Output len=1024, Kernel scheduler=true2026.02 | 227.69 | |
| SGLangBatch size=4, Output len=4096, Kernel scheduler=true2026.02 | 219.48 | |
| vLLMBatch size=4, Output len=16384, Kernel scheduler=true2026.02 | 218.73 | |
| SGLangBatch size=4, Output len=16384, Kernel scheduler=true2026.02 | 213.94 | |
| DeepFusionKernelBatch size=1, Output len=1024, Kernel scheduler=true2026.02 | 61.22 | |
| DeepFusionKernelBatch size=1, Output len=4096, Kernel scheduler=true2026.02 | 61.08 | |
| SGLangBatch size=1, Output len=4096, Kernel scheduler=true2026.02 | 60.77 | |
| DeepFusionKernelBatch size=1, Output len=16384, Kernel scheduler=true2026.02 | 60.72 | |
| SGLangBatch size=1, Output len=1024, Kernel scheduler=true2026.02 | 60.2 | |
| SGLangBatch size=1, Output len=16384, Kernel scheduler=true2026.02 | 60.18 | |
| vLLMBatch size=1, Output len=4096, Kernel scheduler=true2026.02 | 58.7 | |
| vLLMBatch size=1, Output len=1024, Kernel scheduler=true2026.02 | 58.37 | |
| vLLMBatch size=1, Output len=16384, Kernel scheduler=true2026.02 | 57.98 |