Decoding on Llama 70B 3.1 (inference)
1,410.39ThroughputDeepFusionKernel
Evaluation results
| Method | Links | ||
|---|---|---|---|
| DeepFusionKernelBatch size=64, GPU=A100, Output length=10242026.02 | 1,410.39 | 1.3 | |
| SGLangBatch size=64, GPU=A100, Output length=10242026.02 | 1,391.76 | — | |
| DeepFusionKernelBatch size=32, GPU=A100, Output length=10242026.02 | 847.42 | 3.9 | |
| SGLangBatch size=32, GPU=A100, Output length=10242026.02 | 815.37 | — | |
| DeepFusionKernelBatch size=16, GPU=A100, Output length=10242026.02 | 497.69 | 9.7 | |
| SGLangBatch size=16, GPU=A100, Output length=10242026.02 | 453.75 | — | |
| DeepFusionKernelBatch size=8, GPU=A100, Output length=10242026.02 | 270.74 | 6.1 | |
| SGLangBatch size=8, GPU=A100, Output length=10242026.02 | 255.29 | — | |
| TorchBatch size=64, GPU=A100, Output length=10242026.02 | 169.33 | — | |
| DeepFusionKernelBatch size=4, GPU=A100, Output length=10242026.02 | 140.55 | 4.9 | |
| SGLangBatch size=4, GPU=A100, Output length=10242026.02 | 134.02 | — | |
| TorchBatch size=32, GPU=A100, Output length=10242026.02 | 86.43 | — | |
| DeepFusionKernelBatch size=2, GPU=A100, Output length=10242026.02 | 72.6 | 7.7 | |
| SGLangBatch size=2, GPU=A100, Output length=10242026.02 | 67.43 | — | |
| TorchBatch size=16, GPU=A100, Output length=10242026.02 | 40.57 | — | |
| DeepFusionKernelBatch size=1, GPU=A100, Output length=10242026.02 | 37.66 | 5.7 | |
| SGLangBatch size=1, GPU=A100, Output length=10242026.02 | 35.64 | — | |
| TorchBatch size=8, GPU=A100, Output length=10242026.02 | 22.98 | — | |
| TorchBatch size=4, GPU=A100, Output length=10242026.02 | 11.83 | — | |
| TorchBatch size=2, GPU=A100, Output length=10242026.02 | 5.65 | — | |
| TorchBatch size=1, GPU=A100, Output length=10242026.02 | 2.97 | — |