Attention Operator Latency on LLaMA-2 Chat 7B
0.075Attention Latency (ms)CPE-16
Evaluation Results
| Method | Links | |
|---|---|---|
| CPE-16Batch Size (BS)=16, Sequence Length (Seqlen)=1k, block size s=162026.02 | 0.075 | |
| CPE-16Batch Size (BS)=8, Sequence Length (Seqlen)=1k, block size s=162026.02 | 0.078 | |
| CIS-16Batch Size (BS)=16, Sequence Length (Seqlen)=1k, block size s=162026.02 | 0.081 | |
| CIS-16Batch Size (BS)=8, Sequence Length (Seqlen)=1k, block size s=162026.02 | 0.083 | |
| H2OBatch Size (BS)=8, Sequence Length (Seqlen)=1k2026.02 | 0.088 | |
| H2OBatch Size (BS)=16, Sequence Length (Seqlen)=1k2026.02 | 0.089 | |
| CPE-8Batch Size (BS)=16, Sequence Length (Seqlen)=1k, block size s=82026.02 | 0.089 | |
| HShare-1Batch Size (BS)=8, Sequence Length (Seqlen)=1k, ratio=1/2-1/2-1/22026.02 | 0.09 | |
| CPE-8Batch Size (BS)=8, Sequence Length (Seqlen)=1k, block size s=82026.02 | 0.092 | |
| H2OBatch Size (BS)=8, Sequence Length (Seqlen)=2k2026.02 | 0.093 | |
| HShare-1Batch Size (BS)=16, Sequence Length (Seqlen)=1k, ratio=1/2-1/2-1/22026.02 | 0.093 | |
| CPE-16Batch Size (BS)=8, Sequence Length (Seqlen)=2k, block size s=162026.02 | 0.095 | |
| CIS-16Batch Size (BS)=8, Sequence Length (Seqlen)=2k, block size s=162026.02 | 0.106 | |
| H2OBatch Size (BS)=16, Sequence Length (Seqlen)=2k2026.02 | 0.11 | |
| CIS-8Batch Size (BS)=16, Sequence Length (Seqlen)=1k, block size s=82026.02 | 0.113 | |
| CIS-8Batch Size (BS)=8, Sequence Length (Seqlen)=1k, block size s=82026.02 | 0.116 | |
| HShare-1Batch Size (BS)=8, Sequence Length (Seqlen)=2k, ratio=1/2-1/2-1/22026.02 | 0.12 | |
| HShare-0Batch Size (BS)=16, Sequence Length (Seqlen)=1k, ratio=3/4-3/4-1/22026.02 | 0.12 | |
| HShare-0Batch Size (BS)=8, Sequence Length (Seqlen)=1k, ratio=3/4-3/4-1/22026.02 | 0.124 | |
| DSBatch Size (BS)=8, Sequence Length (Seqlen)=1k2026.02 | 0.141 | |
| CPE-8Batch Size (BS)=8, Sequence Length (Seqlen)=2k, block size s=82026.02 | 0.144 | |
| CIS-8Batch Size (BS)=8, Sequence Length (Seqlen)=2k, block size s=82026.02 | 0.147 | |
| HShare-0Batch Size (BS)=8, Sequence Length (Seqlen)=2k, ratio=3/4-3/4-1/22026.02 | 0.16 | |
| CPE-16Batch Size (BS)=16, Sequence Length (Seqlen)=2k, block size s=162026.02 | 0.165 | |
| CIS-16Batch Size (BS)=16, Sequence Length (Seqlen)=2k, block size s=162026.02 | 0.178 | |
| HShare-1Batch Size (BS)=16, Sequence Length (Seqlen)=2k, ratio=1/2-1/2-1/22026.02 | 0.19 | |
| CPE-8Batch Size (BS)=16, Sequence Length (Seqlen)=2k, block size s=82026.02 | 0.196 | |
| QuestBatch Size (BS)=8, Sequence Length (Seqlen)=1k2026.02 | 0.2 | |
| CIS-8Batch Size (BS)=16, Sequence Length (Seqlen)=2k, block size s=82026.02 | 0.214 | |
| FlashBatch Size (BS)=8, Sequence Length (Seqlen)=1k2026.02 | 0.23 | |
| HShare-0Batch Size (BS)=16, Sequence Length (Seqlen)=2k, ratio=3/4-3/4-1/22026.02 | 0.23 | |
| DSBatch Size (BS)=16, Sequence Length (Seqlen)=1k2026.02 | 0.233 | |
| DSBatch Size (BS)=8, Sequence Length (Seqlen)=2k2026.02 | 0.241 | |
| QuestBatch Size (BS)=16, Sequence Length (Seqlen)=1k2026.02 | 0.28 | |
| DSBatch Size (BS)=16, Sequence Length (Seqlen)=2k2026.02 | 0.422 | |
| FlashBatch Size (BS)=16, Sequence Length (Seqlen)=1k2026.02 | 0.44 | |
| QuestBatch Size (BS)=8, Sequence Length (Seqlen)=2k2026.02 | 0.46 | |
| CPE-16Batch Size (BS)=8, Sequence Length (Seqlen)=4k, block size s=162026.02 | 0.469 | |
| H2OBatch Size (BS)=8, Sequence Length (Seqlen)=4k2026.02 | 0.47 | |
| CIS-16Batch Size (BS)=8, Sequence Length (Seqlen)=4k, block size s=162026.02 | 0.501 | |
| CPE-8Batch Size (BS)=8, Sequence Length (Seqlen)=4k, block size s=82026.02 | 0.506 | |
| HShare-1Batch Size (BS)=8, Sequence Length (Seqlen)=4k, ratio=1/2-1/2-1/22026.02 | 0.53 | |
| CIS-8Batch Size (BS)=8, Sequence Length (Seqlen)=4k, block size s=82026.02 | 0.534 | |
| HShare-0Batch Size (BS)=8, Sequence Length (Seqlen)=4k, ratio=3/4-3/4-1/22026.02 | 0.57 | |
| DSBatch Size (BS)=8, Sequence Length (Seqlen)=4k2026.02 | 0.733 | |
| QuestBatch Size (BS)=16, Sequence Length (Seqlen)=2k2026.02 | 0.77 | |
| CPE-16Batch Size (BS)=16, Sequence Length (Seqlen)=4k, block size s=162026.02 | 0.821 | |
| FlashBatch Size (BS)=8, Sequence Length (Seqlen)=2k2026.02 | 0.83 | |
| QuestBatch Size (BS)=8, Sequence Length (Seqlen)=4k2026.02 | 0.85 | |
| H2OBatch Size (BS)=16, Sequence Length (Seqlen)=4k2026.02 | 0.85 | |
| CIS-16Batch Size (BS)=16, Sequence Length (Seqlen)=4k, block size s=162026.02 | 0.869 | |
| CPE-8Batch Size (BS)=16, Sequence Length (Seqlen)=4k, block size s=82026.02 | 0.904 | |
| CIS-8Batch Size (BS)=16, Sequence Length (Seqlen)=4k, block size s=82026.02 | 0.953 | |
| HShare-1Batch Size (BS)=16, Sequence Length (Seqlen)=4k, ratio=1/2-1/2-1/22026.02 | 0.99 | |
| HShare-0Batch Size (BS)=16, Sequence Length (Seqlen)=4k, ratio=3/4-3/4-1/22026.02 | 1.041 | |
| DSBatch Size (BS)=16, Sequence Length (Seqlen)=4k2026.02 | 1.35 | |
| FlashBatch Size (BS)=8, Sequence Length (Seqlen)=4k2026.02 | 1.63 | |
| FlashBatch Size (BS)=16, Sequence Length (Seqlen)=2k2026.02 | 1.63 | |
| QuestBatch Size (BS)=16, Sequence Length (Seqlen)=4k2026.02 | 2.21 | |
| FlashBatch Size (BS)=16, Sequence Length (Seqlen)=4k2026.02 | 3.23 |