LLM Inference on Long-Context LLM Inference Decode
0.13Latency (ms)Reuse
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| ReuseSeqlen=8192, Topk%=10%, Batch size=642025.12 | 0.13 | 0.18 | — | — | |
| KascadeSeqlen=8192, Topk%=10%, Batch size=642025.12 | 0.24 | — | 2.91 | 2.95 | |
| FA3Seqlen=8192, Topk%=10%, Batch size=642025.12 | 0.7 | — | — | — | |
| Tilelang (TL)Seqlen=8192, Topk%=10%, Batch size=642025.12 | 0.71 | — | — | — | |
| AnchorSeqlen=8192, Topk%=10%, Batch size=642025.12 | 0.82 | 1.15 | — | — | |
| Anchor layer 0Seqlen=8192, Topk%=10%, Batch size=642025.12 | 0.92 | 1.3 | — | — | |
| KascadeSeqlen=524288, Topk%=10%, Batch size=322025.12 | 5.33 | — | 4.1 | 4.08 | |
| KascadeSeqlen=524288, Topk%=30%, Batch size=322025.12 | 10.17 | — | 2.15 | 2.14 |