Latency Measurement on LLaMA-8B-Instruct Chunked Prefill 3.1 (inference)
423.1Attention Latency (ms)Dense
Evaluation results
| Method | Links | ||
|---|---|---|---|
| DenseGPU=RTX PRO 6000, TP=2, Batch Size=4, Chunk Size=5122026.05 | 423.1 | 2,299.7 | |
| DenseContext Length=8K, Hardware=H200 SXM GPU, TP=2, Batch size=8, Chunk size=10242026.05 | 429.8 | 1,363.2 | |
| FlashPrefillContext Length=8K, Hardware=H200 SXM GPU, TP=2, Batch size=8, Chunk size=10242026.05 | 464.5 | 1,431.9 | |
| FlashPrefillGPU=RTX PRO 6000, TP=2, Batch Size=4, Chunk Size=5122026.05 | 496.6 | 2,294.2 | |
| CompactAttention-FPContext Length=8K, Hardware=H200 SXM GPU, TP=2, Batch size=8, Chunk size=10242026.05 | 531.5 | 1,476.1 | |
| QUOKAContext Length=8K, Hardware=H200 SXM GPU, TP=2, Batch size=8, Chunk size=10242026.05 | 582.3 | 1,580 | |
| CompactAttention-FPGPU=RTX PRO 6000, TP=2, Batch Size=4, Chunk Size=5122026.05 | 596 | 2,345.1 | |
| QUOKAGPU=RTX PRO 6000, TP=2, Batch Size=4, Chunk Size=5122026.05 | 674 | 2,470.4 | |
| CompactAttention-SAContext Length=8K, Hardware=H200 SXM GPU, TP=2, Batch size=8, Chunk size=10242026.05 | 780.2 | 1,788.3 | |
| XAttentionContext Length=8K, Hardware=H200 SXM GPU, TP=2, Batch size=8, Chunk size=10242026.05 | 923.9 | 1,877 | |
| SeerAttentionContext Length=8K, Hardware=H200 SXM GPU, TP=2, Batch size=8, Chunk size=10242026.05 | 1,002.5 | 1,991.7 | |
| SeerAttentionGPU=RTX PRO 6000, TP=2, Batch Size=4, Chunk Size=5122026.05 | 1,012.3 | 2,911.3 | |
| XAttentionGPU=RTX PRO 6000, TP=2, Batch Size=4, Chunk Size=5122026.05 | 1,095.5 | 2,874.8 | |
| FlashPrefillContext Length=16K, Hardware=H200 SXM GPU, TP=2, Batch size=8, Chunk size=10242026.05 | 1,262.5 | 3,212.2 | |
| DenseContext Length=16K, Hardware=H200 SXM GPU, TP=2, Batch size=8, Chunk size=10242026.05 | 1,272.9 | 3,129 | |
| CompactAttention-FPContext Length=16K, Hardware=H200 SXM GPU, TP=2, Batch size=8, Chunk size=10242026.05 | 1,274.7 | 3,165.8 | |
| CompactAttention-SAGPU=RTX PRO 6000, TP=2, Batch Size=4, Chunk Size=5122026.05 | 1,303 | 3,139.8 | |
| QUOKAContext Length=16K, Hardware=H200 SXM GPU, TP=2, Batch size=8, Chunk size=10242026.05 | 1,516.6 | 3,571.4 | |
| CompactAttention-SAContext Length=16K, Hardware=H200 SXM GPU, TP=2, Batch size=8, Chunk size=10242026.05 | 1,775.7 | 3,745.1 | |
| XAttentionContext Length=16K, Hardware=H200 SXM GPU, TP=2, Batch size=8, Chunk size=10242026.05 | 2,600.1 | 4,536.7 | |
| SeerAttentionContext Length=16K, Hardware=H200 SXM GPU, TP=2, Batch size=8, Chunk size=10242026.05 | 3,073.4 | 5,029.2 | |
| CompactAttention-FPContext Length=32K, Hardware=H200 SXM GPU, TP=2, Batch size=8, Chunk size=10242026.05 | 3,292 | 7,006.8 | |
| FlashPrefillContext Length=32K, Hardware=H200 SXM GPU, TP=2, Batch size=8, Chunk size=10242026.05 | 3,438.8 | 7,276.7 | |
| CompactAttention-FPGPU=RTX PRO 6000, TP=2, Batch Size=4, Chunk Size=5122026.05 | 3,474.8 | 10,691.7 | |
| DenseContext Length=32K, Hardware=H200 SXM GPU, TP=2, Batch size=8, Chunk size=10242026.05 | 4,173.8 | 7,911.3 | |
| CompactAttention-SAContext Length=32K, Hardware=H200 SXM GPU, TP=2, Batch size=8, Chunk size=10242026.05 | 4,393.8 | 8,125.8 | |
| FlashPrefillGPU=RTX PRO 6000, TP=2, Batch Size=4, Chunk Size=5122026.05 | 4,547.7 | 12,088.4 | |
| QUOKAContext Length=32K, Hardware=H200 SXM GPU, TP=2, Batch size=8, Chunk size=10242026.05 | 4,629.7 | 8,482.5 | |
| QUOKAGPU=RTX PRO 6000, TP=2, Batch Size=4, Chunk Size=5122026.05 | 5,196.5 | 13,116.8 | |
| DenseGPU=RTX PRO 6000, TP=2, Batch Size=4, Chunk Size=5122026.05 | 5,213.3 | 12,421.8 | |
| CompactAttention-SAGPU=RTX PRO 6000, TP=2, Batch Size=4, Chunk Size=5122026.05 | 5,584.4 | 12,706.7 | |
| SeerAttentionGPU=RTX PRO 6000, TP=2, Batch Size=4, Chunk Size=5122026.05 | 6,171.8 | 13,620 | |
| XAttentionContext Length=32K, Hardware=H200 SXM GPU, TP=2, Batch size=8, Chunk size=10242026.05 | 7,729.7 | 11,649.8 | |
| XAttentionGPU=RTX PRO 6000, TP=2, Batch Size=4, Chunk Size=5122026.05 | 7,770.5 | 14,896.8 | |
| CompactAttention-FPContext Length=64K, Hardware=H200 SXM GPU, TP=2, Batch size=8, Chunk size=10242026.05 | 8,292 | 15,990 | |
| SeerAttentionContext Length=32K, Hardware=H200 SXM GPU, TP=2, Batch size=8, Chunk size=10242026.05 | 9,478.1 | 13,671 | |
| FlashPrefillContext Length=64K, Hardware=H200 SXM GPU, TP=2, Batch size=8, Chunk size=10242026.05 | 9,489.7 | 17,355.7 | |
| CompactAttention-SAContext Length=64K, Hardware=H200 SXM GPU, TP=2, Batch size=8, Chunk size=10242026.05 | 10,895.6 | 18,453.3 | |
| DenseContext Length=64K, Hardware=H200 SXM GPU, TP=2, Batch size=8, Chunk size=10242026.05 | 14,810.9 | 22,264 | |
| QUOKAContext Length=64K, Hardware=H200 SXM GPU, TP=2, Batch size=8, Chunk size=10242026.05 | 15,012.2 | 22,451.8 | |
| CompactAttention-FPContext Length=128K, Hardware=H200 SXM GPU, TP=2, Batch size=8, Chunk size=10242026.05 | 20,489.7 | 36,265.4 | |
| XAttentionContext Length=64K, Hardware=H200 SXM GPU, TP=2, Batch size=8, Chunk size=10242026.05 | 24,777.8 | 32,093.5 | |
| FlashPrefillContext Length=128K, Hardware=H200 SXM GPU, TP=2, Batch size=8, Chunk size=10242026.05 | 24,886.5 | 41,152.3 | |
| SeerAttentionContext Length=64K, Hardware=H200 SXM GPU, TP=2, Batch size=8, Chunk size=10242026.05 | 27,632.5 | 36,406.8 | |
| CompactAttention-SAContext Length=128K, Hardware=H200 SXM GPU, TP=2, Batch size=8, Chunk size=10242026.05 | 29,468.1 | 45,366.8 | |
| QUOKAContext Length=128K, Hardware=H200 SXM GPU, TP=2, Batch size=8, Chunk size=10242026.05 | 51,292.4 | 66,029.1 | |
| DenseContext Length=128K, Hardware=H200 SXM GPU, TP=2, Batch size=8, Chunk size=10242026.05 | 55,678.4 | 71,015.1 | |
| SeerAttentionContext Length=128K, Hardware=H200 SXM GPU, TP=2, Batch size=8, Chunk size=10242026.05 | 85,852 | 105,207.6 | |
| XAttentionContext Length=128K, Hardware=H200 SXM GPU, TP=2, Batch size=8, Chunk size=10242026.05 | 86,821.4 | 103,870.2 |