LLM Prefill Throughput on LLM Prefill Workload
175,000Prefill Throughput (cuBLASLt)SlideSparse
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| SlideSparseGPU=H100, Model=Llama3.2-1B, Batchsize M=327682026.03 | 175,000 | 1.3 | 1.12 | 1.11 | 1.04 | |
| SlideSparseGPU=H100, Model=Llama3.2-1B, Batchsize M=163842026.03 | 168,000 | 1.2 | 1.12 | 1.12 | 1.02 | |
| SlideSparseGPU=H100, Model=Llama3.2-1B, Batchsize M=81922026.03 | 159,000 | 1.09 | 1.02 | 0.96 | 0.95 | |
| SlideSparseGPU=H100, Model=Llama3.2-1B, Batchsize M=40962026.03 | 131,000 | 0.88 | 0.86 | 0.87 | 0.86 | |
| SlideSparseGPU=A100, Model=Llama3.2-1B, Batchsize M=327682026.03 | 107,000 | 1.5 | 1.29 | 1.24 | 1.12 | |
| SlideSparseGPU=A100, Model=Llama3.2-1B, Batchsize M=163842026.03 | 106,000 | 1.48 | 1.28 | 1.24 | 1.12 | |
| SlideSparseGPU=A100, Model=Llama3.2-1B, Batchsize M=81922026.03 | 103,000 | 1.37 | 1.21 | 1.23 | 1.11 | |
| Dense Softmax 1.3BGPU=H200, SeqLen=4,096, Batch Size=1, Precision=bf16, Number of Parameters=1.3B2026.05 | 102,460 | — | — | — | — | |
| Dense Softmax 1.3BGPU=H100 SXM 80GB, SeqLen=4,096, Batch Size=1, Precision=bf16, Number of Parameters=1.3B2026.05 | 98,476 | — | — | — | — | |
| SlideSparseGPU=RTX4090, Model=Llama3.2-1B, Batchsize M=81922026.03 | 91,400 | 1.24 | 1.14 | 1.1 | 1.05 | |
| SlideSparseGPU=RTX4090, Model=Llama3.2-1B, Batchsize M=163842026.03 | 90,900 | 1.3 | 1.16 | 1.13 | 1.06 | |
| SlideSparseGPU=RTX4090, Model=Llama3.2-1B, Batchsize M=327682026.03 | 90,900 | 1.3 | 1.17 | 1.13 | 1.06 | |
| SlideSparseGPU=A100, Model=Llama3.2-1B, Batchsize M=40962026.03 | 90,300 | 0.88 | 0.83 | 0.88 | 0.87 | |
| Dense Softmax 1.3BGPU=H100 SXM 80GB, SeqLen=32,768, Batch Size=1, Precision=bf16, Number of Parameters=1.3B2026.05 | 71,925 | — | — | — | — | |
| Dense Softmax 1.3BGPU=H200, SeqLen=32,768, Batch Size=1, Precision=bf16, Number of Parameters=1.3B2026.05 | 71,912 | — | — | — | — | |
| SlideSparseGPU=RTX4090, Model=Llama3.2-1B, Batchsize M=40962026.03 | 70,200 | 0.92 | 0.9 | 0.93 | 0.91 | |
| SlideSparseGPU=H100, Model=Llama3.2-1B, Batchsize M=20482026.03 | 68,400 | 0.93 | 0.83 | 0.9 | 0.93 | |
| SlideSparseGPU=A100, Model=BitNet-2B, Batchsize M=327682026.03 | 54,600 | 1.54 | 1.27 | 1.26 | 1.21 | |
| SlideSparseGPU=RTX4090, Model=BitNet-2B, Batchsize M=327682026.03 | 54,500 | 1.31 | 1.11 | 1.11 | 1.08 | |
| SlideSparseGPU=A100, Model=BitNet-2B, Batchsize M=163842026.03 | 54,300 | 1.53 | 1.26 | 1.26 | 1.21 | |
| SlideSparseGPU=RTX4090, Model=BitNet-2B, Batchsize M=40962026.03 | 54,000 | 1.1 | 1.1 | 1.12 | 1.11 | |
| SlideSparseGPU=RTX4090, Model=BitNet-2B, Batchsize M=81922026.03 | 53,600 | 1.31 | 1.06 | 1.05 | 1.04 | |
| SlideSparseGPU=RTX4090, Model=BitNet-2B, Batchsize M=163842026.03 | 53,200 | 1.3 | 1.07 | 1.14 | 1.07 | |
| SlideSparseGPU=A100, Model=BitNet-2B, Batchsize M=81922026.03 | 52,500 | 1.51 | 1.26 | 1.26 | 1.21 | |
| Dense Softmax 1.3BGPU=H200, SeqLen=65,536, Batch Size=1, Precision=bf16, Number of Parameters=1.3B2026.05 | 51,860 | — | — | — | — | |
| Dense Softmax 1.3BGPU=H100 SXM 80GB, SeqLen=65,536, Batch Size=1, Precision=bf16, Number of Parameters=1.3B2026.05 | 51,731 | — | — | — | — | |
| SlideSparseGPU=A100, Model=BitNet-2B, Batchsize M=40962026.03 | 50,900 | 1.48 | 1.26 | 1.26 | 1.21 | |
| SlideSparseGPU=A100, Model=Llama3.2-1B, Batchsize M=20482026.03 | 47,900 | 0.86 | 0.81 | 0.85 | 0.88 | |
| Dense Softmax 1.3BGPU=A100 SXM4 80GB, SeqLen=4,096, Batch Size=1, Precision=bf16, Number of Parameters=1.3B2026.05 | 43,585 | — | — | — | — | |
| SlideSparseGPU=A100, Model=Llama3.2-3B, Batchsize M=327682026.03 | 42,800 | 1.62 | 1.37 | 1.29 | 1.2 | |
| SlideSparseGPU=A100, Model=Llama3.2-3B, Batchsize M=163842026.03 | 42,700 | 1.61 | 1.37 | 1.29 | 1.2 | |
| SlideSparseGPU=A100, Model=Llama3.2-3B, Batchsize M=81922026.03 | 42,100 | 1.59 | 1.36 | 1.27 | 1.19 | |
| SlideSparseGPU=A100, Model=Llama3.2-3B, Batchsize M=40962026.03 | 40,900 | 1.59 | 1.36 | 1.28 | 1.2 | |
| SlideSparseGPU=RTX4090, Model=Llama3.2-3B, Batchsize M=40962026.03 | 40,400 | 1.3 | 1.2 | 1.11 | 1.09 | |
| SlideSparseGPU=RTX4090, Model=Llama3.2-3B, Batchsize M=327682026.03 | 39,800 | 1.34 | 1.18 | 1.13 | 1.08 | |
| SlideSparseGPU=RTX4090, Model=Llama3.2-3B, Batchsize M=163842026.03 | 39,600 | 1.36 | 1.18 | 1.12 | 1.09 | |
| SlideSparseGPU=RTX4090, Model=Llama3.2-3B, Batchsize M=81922026.03 | 39,400 | 1.35 | 1.19 | 1.13 | 1.09 | |
| SlideSparseGPU=A100, Model=Llama3.2-3B, Batchsize M=20482026.03 | 37,700 | 1.13 | 1.15 | 1.13 | 1.14 | |
| SlideSparseGPU=A100, Model=BitNet-2B, Batchsize M=20482026.03 | 37,300 | 1.03 | 1.08 | 1.09 | 1.09 | |
| MoL Hybrid 2.08BGPU=H200, SeqLen=32,768, Batch Size=1, Precision=bf16, Number of Parameters=2.08B2026.05 | 36,911 | — | — | — | — | |
| SlideSparseGPU=RTX4090, Model=Llama3.2-1B, Batchsize M=20482026.03 | 36,500 | 0.92 | 0.9 | 0.88 | 0.88 | |
| MoL Hybrid 2.08BGPU=H200, SeqLen=65,536, Batch Size=1, Precision=bf16, Number of Parameters=2.08B, Path-selection protocol=sparse2026.05 | 36,292 | — | — | — | — | |
| SlideSparseGPU=H100, Model=Llama3.2-1B, Batchsize M=10242026.03 | 35,600 | 0.88 | 0.9 | 0.87 | 0.9 | |
| MoL Hybrid 2.08BGPU=H200, SeqLen=65,536, Batch Size=1, Precision=bf16, Number of Parameters=2.08B, Path-selection protocol=batched2026.05 | 35,244 | — | — | — | — | |
| MoL Hybrid 2.08BGPU=H100 SXM 80GB, SeqLen=32,768, Batch Size=1, Precision=bf16, Number of Parameters=2.08B2026.05 | 33,579 | — | — | — | — | |
| MoL Hybrid 2.08BGPU=H100 SXM 80GB, SeqLen=65,536, Batch Size=1, Precision=bf16, Number of Parameters=2.08B2026.05 | 33,559 | — | — | — | — | |
| MoL Hybrid 2.08BGPU=H200, SeqLen=131,072, Batch Size=1, Precision=bf16, Number of Parameters=2.08B, Path-selection protocol=sparse2026.05 | 33,008 | — | — | — | — | |
| SlideSparseGPU=RTX4090, Model=BitNet-2B, Batchsize M=20482026.03 | 31,500 | 1.12 | 1.13 | 1.07 | 1.13 | |
| Dense Softmax 1.3BGPU=H200, SeqLen=131,072, Batch Size=1, Precision=bf16, Number of Parameters=1.3B2026.05 | 31,348 | — | — | — | — | |
| Dense Softmax 1.3BGPU=H100 SXM 80GB, SeqLen=131,072, Batch Size=1, Precision=bf16, Number of Parameters=1.3B2026.05 | 31,299 | — | — | — | — | |
| MoL Hybrid 2.08BGPU=H100 SXM 80GB, SeqLen=131,072, Batch Size=1, Precision=bf16, Number of Parameters=2.08B, Path-selection protocol=sparse2026.05 | 30,819 | — | — | — | — | |
| Dense Softmax 1.3BGPU=A100 SXM4 80GB, SeqLen=32,768, Batch Size=1, Precision=bf16, Number of Parameters=1.3B2026.05 | 28,472 | — | — | — | — | |
| SlideSparseGPU=RTX4090, Model=Llama3.2-3B, Batchsize M=20482026.03 | 27,800 | 1.09 | 1.12 | 1.11 | 1.06 | |
| MoL Hybrid 2.08BGPU=H200, SeqLen=262,144, Batch Size=1, Precision=bf16, Number of Parameters=2.08B, Path-selection protocol=sparse2026.05 | 26,105 | — | — | — | — | |
| MoL Hybrid 2.08BGPU=H100 SXM 80GB, SeqLen=262,144, Batch Size=1, Precision=bf16, Number of Parameters=2.08B, Path-selection protocol=sparse2026.05 | 24,841 | — | — | — | — | |
| SlideSparseGPU=A100, Model=Llama3.2-1B, Batchsize M=10242026.03 | 24,600 | 0.85 | 0.71 | 0.84 | 0.84 | |
| SlideSparseGPU=RTX4090, Model=Qwen2.5-7B, Batchsize M=327682026.03 | 21,400 | 0.41 | 0.31 | 0.27 | 0.25 | |
| SlideSparseGPU=RTX4090, Model=Qwen2.5-7B, Batchsize M=163842026.03 | 21,300 | 1.39 | 1.18 | 0.58 | 0.4 | |
| SlideSparseGPU=RTX4090, Model=Qwen2.5-7B, Batchsize M=81922026.03 | 21,200 | 1.39 | 1.18 | 1.11 | 1.08 | |
| SlideSparseGPU=A100, Model=Qwen2.5-7B, Batchsize M=163842026.03 | 20,900 | 1.75 | 1.41 | 1.34 | 1.27 | |
| SlideSparseGPU=A100, Model=Qwen2.5-7B, Batchsize M=81922026.03 | 20,800 | 1.72 | 1.4 | 1.32 | 1.26 | |
| SlideSparseGPU=A100, Model=Qwen2.5-7B, Batchsize M=327682026.03 | 20,800 | 1.75 | 1.41 | 1.34 | 1.26 | |
| SlideSparseGPU=RTX4090, Model=Qwen2.5-7B, Batchsize M=40962026.03 | 20,700 | 1.42 | 1.21 | 1.13 | 1.08 | |
| MoL all-softmax 2.08BGPU=RTX 3090, SeqLen=4,096, Batch Size=1, Precision=bf16, Number of Parameters=2.08B, Randomly Initialized=true2026.05 | 20,591 | — | — | — | — | |
| Dense DeltaNet 1.3BGPU=RTX 3090, SeqLen=32,768, Batch Size=1, Precision=bf16, Number of Parameters=1.3B, Randomly Initialized=true2026.05 | 20,502 | — | — | — | — | |
| SlideSparseGPU=A100, Model=Llama3.2-3B, Batchsize M=10242026.03 | 20,300 | 1.04 | 1.04 | 1.05 | 1.02 | |
| SlideSparseGPU=A100, Model=Qwen2.5-7B, Batchsize M=40962026.03 | 20,300 | 1.7 | 1.38 | 1.3 | 1.24 | |
| SlideSparseGPU=RTX4090, Model=Qwen2.5-7B, Batchsize M=20482026.03 | 20,300 | 1.38 | 1.18 | 1.12 | 1.09 | |
| Dense Softmax 1.3BGPU=A100 SXM4 80GB, SeqLen=65,536, Batch Size=1, Precision=bf16, Number of Parameters=1.3B2026.05 | 19,495 | — | — | — | — | |
| MoL Hybrid 2.08BGPU=RTX 3090, SeqLen=32,768, Batch Size=1, Precision=bf16, Number of Parameters=2.08B2026.05 | 19,062 | — | — | — | — | |
| SlideSparseGPU=A100, Model=Qwen2.5-7B, Batchsize M=20482026.03 | 18,800 | 1.63 | 1.34 | 1.26 | 1.2 | |
| SlideSparseGPU=RTX4090, Model=Llama3.2-1B, Batchsize M=10242026.03 | 18,700 | 0.85 | 0.88 | 0.89 | 0.87 | |
| SlideSparseGPU=A100, Model=BitNet-2B, Batchsize M=10242026.03 | 18,300 | 1.03 | 1.06 | 1.13 | 1.11 | |
| SlideSparseGPU=H100, Model=Llama3.2-1B, Batchsize M=5122026.03 | 18,200 | 0.87 | 0.9 | 0.85 | 0.91 | |
| Dense Softmax 1.3BGPU=RTX 3090, SeqLen=4,096, Batch Size=1, Precision=bf16, Number of Parameters=1.3B2026.05 | 18,041 | — | — | — | — | |
| SlideSparseGPU=A100, Model=Qwen2.5-7B, Batchsize M=10242026.03 | 18,000 | 1.24 | 1.25 | 1.18 | 1.14 | |
| Dense DeltaNet 1.3BGPU=RTX 3090, SeqLen=4,096, Batch Size=1, Precision=bf16, Number of Parameters=1.3B, Randomly Initialized=true2026.05 | 17,962 | — | — | — | — | |
| Dense Softmax 1.3BGPU=H100 SXM 80GB, SeqLen=262,144, Batch Size=1, Precision=bf16, Number of Parameters=1.3B2026.05 | 17,495 | — | — | — | — | |
| Dense Softmax 1.3BGPU=H200, SeqLen=262,144, Batch Size=1, Precision=bf16, Number of Parameters=1.3B2026.05 | 16,973 | — | — | — | — | |
| MoL Hybrid 2.08BGPU=A100 SXM4 80GB, SeqLen=32,768, Batch Size=1, Precision=bf16, Number of Parameters=2.08B2026.05 | 16,650 | — | — | — | — | |
| MoL Hybrid 2.08BGPU=H100 SXM 80GB, SeqLen=4,096, Batch Size=1, Precision=bf16, Number of Parameters=2.08B2026.05 | 16,459 | — | — | — | — | |
| MoL all-softmax 2.08BGPU=RTX 3090, SeqLen=32,768, Batch Size=1, Precision=bf16, Number of Parameters=2.08B, Randomly Initialized=true2026.05 | 16,424 | — | — | — | — | |
| SlideSparseGPU=RTX4090, Model=Qwen2.5-7B, Batchsize M=10242026.03 | 16,300 | 1.07 | 1.09 | 1.05 | 1.04 | |
| MoL Hybrid 2.08BGPU=RTX 3090, SeqLen=4,096, Batch Size=1, Precision=bf16, Number of Parameters=2.08B2026.05 | 16,005 | — | — | — | — | |
| SlideSparseGPU=RTX4090, Model=BitNet-2B, Batchsize M=10242026.03 | 16,000 | 1.11 | 1.06 | 1.07 | 1.04 | |
| MoL Hybrid 2.08BGPU=A100 SXM4 80GB, SeqLen=65,536, Batch Size=1, Precision=bf16, Number of Parameters=2.08B2026.05 | 15,968 | — | — | — | — | |
| SlideSparseGPU=H100, Model=BitNet-2B, Batchsize M=5122026.03 | 14,900 | 1.1 | 1.15 | 1.15 | 1.09 | |
| MoL Hybrid 2.08BGPU=H200, SeqLen=4,096, Batch Size=1, Precision=bf16, Number of Parameters=2.08B2026.05 | 14,174 | — | — | — | — | |
| MoL Hybrid 2.08BGPU=A100 SXM4 80GB, SeqLen=131,072, Batch Size=1, Precision=bf16, Number of Parameters=2.08B, Path-selection protocol=sparse2026.05 | 14,115 | — | — | — | — | |
| SlideSparseGPU=RTX4090, Model=Llama3.2-3B, Batchsize M=10242026.03 | 13,800 | 1.01 | 1.12 | 1.07 | 1.11 | |
| SlideSparseGPU=A100, Model=Llama3.2-1B, Batchsize M=5122026.03 | 11,900 | 0.88 | 0.89 | 0.84 | 0.88 | |
| Dense Softmax 1.3BGPU=A100 SXM4 80GB, SeqLen=131,072, Batch Size=1, Precision=bf16, Number of Parameters=1.3B2026.05 | 11,779 | — | — | — | — | |
| SlideSparseGPU=RTX4090, Model=Qwen2.5-14B, Batchsize M=40962026.03 | 11,400 | 1.4 | 1.19 | 1.12 | — | |
| SlideSparseGPU=RTX4090, Model=Qwen2.5-14B, Batchsize M=81922026.03 | 11,300 | 1.4 | 1.18 | 1.12 | — | |
| SlideSparseGPU=RTX4090, Model=Qwen2.5-14B, Batchsize M=163842026.03 | 11,300 | 1.4 | 1.19 | — | — | |
| SlideSparseGPU=RTX4090, Model=Qwen2.5-14B, Batchsize M=20482026.03 | 11,100 | 1.43 | 1.21 | 1.14 | — | |
| Dense Softmax 1.3BGPU=RTX 3090, SeqLen=32,768, Batch Size=1, Precision=bf16, Number of Parameters=1.3B2026.05 | 10,808 | — | — | — | — | |
| SlideSparseGPU=A100, Model=Qwen2.5-14B, Batchsize M=327682026.03 | 10,700 | 1.77 | 1.42 | 1.33 | 1.26 | |
| SlideSparseGPU=A100, Model=Qwen2.5-14B, Batchsize M=81922026.03 | 10,600 | 1.77 | 1.43 | 1.34 | 1.27 | |
| SlideSparseGPU=A100, Model=Qwen2.5-14B, Batchsize M=163842026.03 | 10,600 | 1.77 | 1.43 | 1.34 | 1.27 |