Prefill Latency Measurement on LLaMA 8B Instruct 128K context length 3.1
24,055.4Attention Latency (ms)CompactAttention-FP
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| CompactAttention-FPGPU=RTX PRO 6000, TP=2, Batch Size=4, Chunk Size=5122026.05 | 24,055.4 | 54,130.6 | |
| CompactAttention-SAGPU=RTX PRO 6000, TP=2, Batch Size=4, Chunk Size=5122026.05 | 32,846 | 64,126.7 | |
| FlashPrefillGPU=RTX PRO 6000, TP=2, Batch Size=4, Chunk Size=5122026.05 | 43,750.5 | 80,108.5 | |
| SeerAttentionGPU=RTX PRO 6000, TP=2, Batch Size=4, Chunk Size=5122026.05 | 49,839.8 | 81,774.4 | |
| QUOKAGPU=RTX PRO 6000, TP=2, Batch Size=4, Chunk Size=5122026.05 | 56,475.3 | 85,375.3 | |
| DenseGPU=RTX PRO 6000, TP=2, Batch Size=4, Chunk Size=5122026.05 | 65,971.1 | 94,944.3 | |
| XAttentionGPU=RTX PRO 6000, TP=2, Batch Size=4, Chunk Size=5122026.05 | 82,617 | 113,016.5 |