Prefill Latency Measurement on LLaMA 8B-Instruct 64K context length 3.1
8,945.9Attention LatencyCompactAttention-FP
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| CompactAttention-FPGPU=RTX PRO 6000, TP=2, Batch Size=4, Chunk Size=5122026.05 | 8,945.9 | 23,707.8 | |
| CompactAttention-SAGPU=RTX PRO 6000, TP=2, Batch Size=4, Chunk Size=5122026.05 | 13,823.5 | 28,960.4 | |
| FlashPrefillGPU=RTX PRO 6000, TP=2, Batch Size=4, Chunk Size=5122026.05 | 14,381.8 | 29,411.7 | |
| QUOKAGPU=RTX PRO 6000, TP=2, Batch Size=4, Chunk Size=5122026.05 | 16,325.4 | 30,588 | |
| SeerAttentionGPU=RTX PRO 6000, TP=2, Batch Size=4, Chunk Size=5122026.05 | 17,078.2 | 32,545.8 | |
| DenseGPU=RTX PRO 6000, TP=2, Batch Size=4, Chunk Size=5122026.05 | 17,838.8 | 32,177.4 | |
| XAttentionGPU=RTX PRO 6000, TP=2, Batch Size=4, Chunk Size=5122026.05 | 24,819.2 | 39,206.6 |