List your company
ResearchFormerly WizWand

LLM Decoding on Llama 70B (H100 GPU Cluster) 3.1

894.32ThroughputDeepFusionKernel
24.5264250.3382476.15701.9618Feb 12, 2026
Updated 8mo ago

Evaluation results

MethodLinks
DeepFusionKernelBatch size=16, Output len=1024, Kernel scheduler=true2026.02894.32
SGLangBatch size=16, Output len=1024, Kernel scheduler=true2026.02878.08
vLLMBatch size=16, Output len=1024, Kernel scheduler=true2026.02865.37
DeepFusionKernelBatch size=16, Output len=4096, Kernel scheduler=true2026.02863.4
vLLMBatch size=16, Output len=4096, Kernel scheduler=true2026.02850.33
SGLangBatch size=16, Output len=4096, Kernel scheduler=true2026.02815.4
DeepFusionKernelBatch size=16, Output len=16384, Kernel scheduler=true2026.02764.57
vLLMBatch size=16, Output len=16384, Kernel scheduler=true2026.02746.42
SGLangBatch size=16, Output len=16384, Kernel scheduler=true2026.02732.18
DeepFusionKernelBatch size=4, Output len=4096, Kernel scheduler=true2026.02234.61
DeepFusionKernelBatch size=4, Output len=1024, Kernel scheduler=true2026.02230.98
SGLangBatch size=4, Output len=1024, Kernel scheduler=true2026.02228.9
DeepFusionKernelBatch size=4, Output len=16384, Kernel scheduler=true2026.02227.73
vLLMBatch size=4, Output len=4096, Kernel scheduler=true2026.02227.7
vLLMBatch size=4, Output len=1024, Kernel scheduler=true2026.02227.69
SGLangBatch size=4, Output len=4096, Kernel scheduler=true2026.02219.48
vLLMBatch size=4, Output len=16384, Kernel scheduler=true2026.02218.73
SGLangBatch size=4, Output len=16384, Kernel scheduler=true2026.02213.94
DeepFusionKernelBatch size=1, Output len=1024, Kernel scheduler=true2026.0261.22
DeepFusionKernelBatch size=1, Output len=4096, Kernel scheduler=true2026.0261.08
SGLangBatch size=1, Output len=4096, Kernel scheduler=true2026.0260.77
DeepFusionKernelBatch size=1, Output len=16384, Kernel scheduler=true2026.0260.72
SGLangBatch size=1, Output len=1024, Kernel scheduler=true2026.0260.2
SGLangBatch size=1, Output len=16384, Kernel scheduler=true2026.0260.18
vLLMBatch size=1, Output len=4096, Kernel scheduler=true2026.0258.7
vLLMBatch size=1, Output len=1024, Kernel scheduler=true2026.0258.37
vLLMBatch size=1, Output len=16384, Kernel scheduler=true2026.0257.98