Prefill Latency Measurement on LLaMA 8B Instruct 16K context length 3.1
1,224Attention Prefill LatencyDense
Evaluation results
| Method | Links | ||
|---|---|---|---|
| DenseGPU=RTX PRO 6000, TP=2, Batch Size=4, Chunk Size=5122026.05 | 1,224 | 4,858 | |
| FlashPrefillGPU=RTX PRO 6000, TP=2, Batch Size=4, Chunk Size=5122026.05 | 1,356.2 | 4,978.7 | |
| CompactAttention-FPGPU=RTX PRO 6000, TP=2, Batch Size=4, Chunk Size=5122026.05 | 1,393.6 | 4,953.2 | |
| QUOKAGPU=RTX PRO 6000, TP=2, Batch Size=4, Chunk Size=5122026.05 | 1,599.5 | 5,156.3 | |
| SeerAttentionGPU=RTX PRO 6000, TP=2, Batch Size=4, Chunk Size=5122026.05 | 2,548.7 | 6,286.4 | |
| CompactAttention-SAGPU=RTX PRO 6000, TP=2, Batch Size=4, Chunk Size=5122026.05 | 2,675.7 | 6,382.9 | |
| XAttentionGPU=RTX PRO 6000, TP=2, Batch Size=4, Chunk Size=5122026.05 | 2,700.5 | 6,243.9 |