Sparse Decode Performance on 128K Context irregular sparse decode
76.77Speedup Factorirregular sparse decode
Evaluation results
| Method | Links | ||
|---|---|---|---|
| irregular sparse decodeBatch size (B)=16, Sparsity Level (S)=500x, Hardware=H100 80GB HBM3, Precision=FP16, Attention Configuration=GQA Hq=32, Hkv=8, D=128, Context Length=128K, Page size=162026.05 | 76.77 | — | |
| irregular sparse decodeBatch size (B)=8, Sparsity Level (S)=500x, Hardware=H100 80GB HBM3, Precision=FP16, Attention Configuration=GQA Hq=32, Hkv=8, D=128, Context Length=128K, Page size=162026.05 | 76.14 | — | |
| irregular sparse decodeBatch size (B)=4, Sparsity Level (S)=500x, Hardware=H100 80GB HBM3, Precision=FP16, Attention Configuration=GQA Hq=32, Hkv=8, D=128, Context Length=128K, Page size=162026.05 | 42.04 | — | |
| irregular sparse decodeBatch size (B)=16, Sparsity Level (S)=200x, Hardware=H100 80GB HBM3, Precision=FP16, Attention Configuration=GQA Hq=32, Hkv=8, D=128, Context Length=128K, Page size=162026.05 | 37.32 | — | |
| irregular sparse decodeBatch size (B)=8, Sparsity Level (S)=200x, Hardware=H100 80GB HBM3, Precision=FP16, Attention Configuration=GQA Hq=32, Hkv=8, D=128, Context Length=128K, Page size=162026.05 | 29.64 | — | |
| irregular sparse decodeBatch size (B)=4, Sparsity Level (S)=200x, Hardware=H100 80GB HBM3, Precision=FP16, Attention Configuration=GQA Hq=32, Hkv=8, D=128, Context Length=128K, Page size=162026.05 | 24.25 | — | |
| irregular sparse decodeBatch size (B)=16, Sparsity Level (S)=100x, Hardware=H100 80GB HBM3, Precision=FP16, Attention Configuration=GQA Hq=32, Hkv=8, D=128, Context Length=128K, Page size=162026.05 | 20.09 | — | |
| irregular sparse decodeBatch size (B)=8, Sparsity Level (S)=100x, Hardware=H100 80GB HBM3, Precision=FP16, Attention Configuration=GQA Hq=32, Hkv=8, D=128, Context Length=128K, Page size=162026.05 | 16.82 | — | |
| irregular sparse decodeBatch size (B)=4, Sparsity Level (S)=100x, Hardware=H100 80GB HBM3, Precision=FP16, Attention Configuration=GQA Hq=32, Hkv=8, D=128, Context Length=128K, Page size=162026.05 | 13.36 | — | |
| irregular sparse decodeBatch size (B)=1, Sparsity Level (S)=500x, Hardware=H100 80GB HBM3, Precision=FP16, Attention Configuration=GQA Hq=32, Hkv=8, D=128, Context Length=128K, Page size=162026.05 | 11.14 | — | |
| irregular sparse decodeBatch size (B)=1, Sparsity Level (S)=200x, Hardware=H100 80GB HBM3, Precision=FP16, Attention Configuration=GQA Hq=32, Hkv=8, D=128, Context Length=128K, Page size=162026.05 | 11.05 | — | |
| irregular sparse decodeBatch size (B)=16, Sparsity Level (S)=50x, Hardware=H100 80GB HBM3, Precision=FP16, Attention Configuration=GQA Hq=32, Hkv=8, D=128, Context Length=128K, Page size=162026.05 | 10.54 | — | |
| irregular sparse decodeBatch size (B)=1, Sparsity Level (S)=100x, Hardware=H100 80GB HBM3, Precision=FP16, Attention Configuration=GQA Hq=32, Hkv=8, D=128, Context Length=128K, Page size=162026.05 | 10.25 | — | |
| irregular sparse decodeBatch size (B)=8, Sparsity Level (S)=50x, Hardware=H100 80GB HBM3, Precision=FP16, Attention Configuration=GQA Hq=32, Hkv=8, D=128, Context Length=128K, Page size=162026.05 | 8.88 | — | |
| irregular sparse decodeBatch size (B)=4, Sparsity Level (S)=50x, Hardware=H100 80GB HBM3, Precision=FP16, Attention Configuration=GQA Hq=32, Hkv=8, D=128, Context Length=128K, Page size=162026.05 | 7.45 | — | |
| irregular sparse decodeBatch size (B)=1, Sparsity Level (S)=50x, Hardware=H100 80GB HBM3, Precision=FP16, Attention Configuration=GQA Hq=32, Hkv=8, D=128, Context Length=128K, Page size=162026.05 | 5.57 | — | |
| irregular sparse decodeBatch size (B)=16, Sparsity Level (S)=20x, Hardware=H100 80GB HBM3, Precision=FP16, Attention Configuration=GQA Hq=32, Hkv=8, D=128, Context Length=128K, Page size=162026.05 | 4.35 | — | |
| irregular sparse decodeBatch size (B)=8, Sparsity Level (S)=20x, Hardware=H100 80GB HBM3, Precision=FP16, Attention Configuration=GQA Hq=32, Hkv=8, D=128, Context Length=128K, Page size=162026.05 | 3.75 | — | |
| irregular sparse decodeBatch size (B)=4, Sparsity Level (S)=20x, Hardware=H100 80GB HBM3, Precision=FP16, Attention Configuration=GQA Hq=32, Hkv=8, D=128, Context Length=128K, Page size=162026.05 | 3.18 | — | |
| irregular sparse decodeBatch size (B)=1, Sparsity Level (S)=20x, Hardware=H100 80GB HBM3, Precision=FP16, Attention Configuration=GQA Hq=32, Hkv=8, D=128, Context Length=128K, Page size=162026.05 | 2.58 | — | |
| irregular sparse decodeBatch size (B)=16, Sparsity Level (S)=10x, Hardware=H100 80GB HBM3, Precision=FP16, Attention Configuration=GQA Hq=32, Hkv=8, D=128, Context Length=128K, Page size=162026.05 | 2.21 | — | |
| irregular sparse decodeBatch size (B)=8, Sparsity Level (S)=10x, Hardware=H100 80GB HBM3, Precision=FP16, Attention Configuration=GQA Hq=32, Hkv=8, D=128, Context Length=128K, Page size=162026.05 | 1.9 | — | |
| irregular sparse decodeBatch size (B)=4, Sparsity Level (S)=10x, Hardware=H100 80GB HBM3, Precision=FP16, Attention Configuration=GQA Hq=32, Hkv=8, D=128, Context Length=128K, Page size=162026.05 | 1.64 | — | |
| irregular sparse decodeBatch size (B)=1, Sparsity Level (S)=10x, Hardware=H100 80GB HBM3, Precision=FP16, Attention Configuration=GQA Hq=32, Hkv=8, D=128, Context Length=128K, Page size=162026.05 | 1.45 | — | |
| irregular sparse decodeBatch size (B)=16, Sparsity Level (S)=4x, Hardware=H100 80GB HBM3, Precision=FP16, Attention Configuration=GQA Hq=32, Hkv=8, D=128, Context Length=128K, Page size=162026.05 | 0.89 | — | |
| irregular sparse decodeBatch size (B)=8, Sparsity Level (S)=4x, Hardware=H100 80GB HBM3, Precision=FP16, Attention Configuration=GQA Hq=32, Hkv=8, D=128, Context Length=128K, Page size=162026.05 | 0.77 | — | |
| irregular sparse decodeBatch size (B)=4, Sparsity Level (S)=4x, Hardware=H100 80GB HBM3, Precision=FP16, Attention Configuration=GQA Hq=32, Hkv=8, D=128, Context Length=128K, Page size=162026.05 | 0.66 | — | |
| irregular sparse decodeBatch size (B)=1, Sparsity Level (S)=4x, Hardware=H100 80GB HBM3, Precision=FP16, Attention Configuration=GQA Hq=32, Hkv=8, D=128, Context Length=128K, Page size=162026.05 | 0.63 | — | |
| irregular sparse decodeBatch size (B)=16, Sparsity Level (S)=2x, Hardware=H100 80GB HBM3, Precision=FP16, Attention Configuration=GQA Hq=32, Hkv=8, D=128, Context Length=128K, Page size=162026.05 | 0.45 | — | |
| irregular sparse decodeBatch size (B)=8, Sparsity Level (S)=2x, Hardware=H100 80GB HBM3, Precision=FP16, Attention Configuration=GQA Hq=32, Hkv=8, D=128, Context Length=128K, Page size=162026.05 | 0.38 | — | |
| irregular sparse decodeBatch size (B)=4, Sparsity Level (S)=2x, Hardware=H100 80GB HBM3, Precision=FP16, Attention Configuration=GQA Hq=32, Hkv=8, D=128, Context Length=128K, Page size=162026.05 | 0.33 | — | |
| irregular sparse decodeBatch size (B)=1, Sparsity Level (S)=2x, Hardware=H100 80GB HBM3, Precision=FP16, Attention Configuration=GQA Hq=32, Hkv=8, D=128, Context Length=128K, Page size=162026.05 | 0.32 | — | |
| FlashInferBatch size (B)=1, Hardware=H100 80GB HBM3, Precision=FP16, Attention Configuration=GQA Hq=32, Hkv=8, D=128, Context Length=128K, Page size=162026.05 | — | 0.19 | |
| FlashInferBatch size (B)=4, Hardware=H100 80GB HBM3, Precision=FP16, Attention Configuration=GQA Hq=32, Hkv=8, D=128, Context Length=128K, Page size=162026.05 | — | 0.72 | |
| FlashInferBatch size (B)=8, Hardware=H100 80GB HBM3, Precision=FP16, Attention Configuration=GQA Hq=32, Hkv=8, D=128, Context Length=128K, Page size=162026.05 | — | 1.5 | |
| FlashInferBatch size (B)=16, Hardware=H100 80GB HBM3, Precision=FP16, Attention Configuration=GQA Hq=32, Hkv=8, D=128, Context Length=128K, Page size=162026.05 | — | 3.08 |