LLM Decoding on Llama 70B 3.1
3,119.55ThroughputDeepFusionKernel
Evaluation results
| Method | Links | ||
|---|---|---|---|
| DeepFusionKernelBatch size=64, GPU=H100, Output length=10242026.02 | 3,119.55 | 3.4 | |
| SGLangBatch size=64, GPU=H100, Output length=10242026.02 | 3,016.4 | — | |
| DeepFusionKernelBatch size=32, GPU=H100, Output length=10242026.02 | 1,635.94 | 5.5 | |
| SGLangBatch size=32, GPU=H100, Output length=10242026.02 | 1,551.33 | — | |
| DeepFusionKernelBatch size=16, GPU=H100, Output length=10242026.02 | 914.62 | 4.2 | |
| SGLangBatch size=16, GPU=H100, Output length=10242026.02 | 878.08 | — | |
| DeepFusionKernelBatch size=16, Output len=1024, Kernel scheduler=true2026.02 | 493.6 | — | |
| DeepFusionKernelBatch size=16, Output len=4096, Kernel scheduler=true2026.02 | 478.66 | — | |
| DeepFusionKernelBatch size=8, GPU=H100, Output length=10242026.02 | 470.63 | 4.2 | |
| vLLMBatch size=16, Output len=1024, Kernel scheduler=true2026.02 | 462.01 | — | |
| SGLangBatch size=16, Output len=1024, Kernel scheduler=true2026.02 | 453.75 | — | |
| SGLangBatch size=16, Output len=4096, Kernel scheduler=true2026.02 | 452.6 | — | |
| SGLangBatch size=8, GPU=H100, Output length=10242026.02 | 451.64 | — | |
| vLLMBatch size=16, Output len=4096, Kernel scheduler=true2026.02 | 448.93 | — | |
| DeepFusionKernelBatch size=16, Output len=16384, Kernel scheduler=true2026.02 | 416.19 | — | |
| SGLangBatch size=16, Output len=16384, Kernel scheduler=true2026.02 | 396.7 | — | |
| vLLMBatch size=16, Output len=16384, Kernel scheduler=true2026.02 | 390.94 | — | |
| DeepFusionKernelBatch size=4, GPU=H100, Output length=10242026.02 | 237.05 | 3.6 | |
| SGLangBatch size=4, GPU=H100, Output length=10242026.02 | 228.9 | — | |
| DeepFusionKernelBatch size=4, Output len=1024, Kernel scheduler=true2026.02 | 140.15 | — | |
| DeepFusionKernelBatch size=4, Output len=4096, Kernel scheduler=true2026.02 | 138.16 | — | |
| SGLangBatch size=4, Output len=1024, Kernel scheduler=true2026.02 | 135.76 | — | |
| SGLangBatch size=4, Output len=4096, Kernel scheduler=true2026.02 | 132.66 | — | |
| DeepFusionKernelBatch size=4, Output len=16384, Kernel scheduler=true2026.02 | 131.75 | — | |
| TorchBatch size=64, GPU=H100, Output length=10242026.02 | 129.81 | — | |
| SGLangBatch size=4, Output len=16384, Kernel scheduler=true2026.02 | 126.81 | — | |
| vLLMBatch size=4, Output len=1024, Kernel scheduler=true2026.02 | 126.1 | — | |
| vLLMBatch size=4, Output len=4096, Kernel scheduler=true2026.02 | 125.1 | — | |
| DeepFusionKernelBatch size=2, GPU=H100, Output length=10242026.02 | 121.13 | 13.2 | |
| vLLMBatch size=4, Output len=16384, Kernel scheduler=true2026.02 | 119.3 | — | |
| TorchBatch size=32, GPU=H100, Output length=10242026.02 | 118.56 | — | |
| SGLangBatch size=2, GPU=H100, Output length=10242026.02 | 107.05 | — | |
| DeepFusionKernelBatch size=1, GPU=H100, Output length=10242026.02 | 62.22 | 3.4 | |
| SGLangBatch size=1, GPU=H100, Output length=10242026.02 | 60.2 | — | |
| TorchBatch size=16, GPU=H100, Output length=10242026.02 | 52.37 | — | |
| DeepFusionKernelBatch size=1, Output len=1024, Kernel scheduler=true2026.02 | 37.57 | — | |
| DeepFusionKernelBatch size=1, Output len=4096, Kernel scheduler=true2026.02 | 37.5 | — | |
| DeepFusionKernelBatch size=1, Output len=16384, Kernel scheduler=true2026.02 | 36.79 | — | |
| SGLangBatch size=1, Output len=1024, Kernel scheduler=true2026.02 | 35.68 | — | |
| SGLangBatch size=1, Output len=4096, Kernel scheduler=true2026.02 | 35.53 | — | |
| SGLangBatch size=1, Output len=16384, Kernel scheduler=true2026.02 | 34.97 | — | |
| TorchBatch size=8, GPU=H100, Output length=10242026.02 | 34.3 | — | |
| vLLMBatch size=1, Output len=4096, Kernel scheduler=true2026.02 | 33.64 | — | |
| vLLMBatch size=1, Output len=1024, Kernel scheduler=true2026.02 | 33.47 | — | |
| vLLMBatch size=1, Output len=16384, Kernel scheduler=true2026.02 | 32.93 | — | |
| TorchBatch size=4, GPU=H100, Output length=10242026.02 | 18.11 | — | |
| TorchBatch size=2, GPU=H100, Output length=10242026.02 | 8.39 | — | |
| TorchBatch size=1, GPU=H100, Output length=10242026.02 | 3.55 | — |