List your company
ResearchFormerly WizWand

Attention Operator Latency on LLaMA-2 Chat 7B

0.075Attention Latency (ms)CPE-16
-0.05120.800651.65252.50435Feb 9, 2026
Updated 8mo ago

Evaluation results

MethodLinks
CPE-16Batch Size (BS)=16, Sequence Length (Seqlen)=1k, block size s=162026.020.075
CPE-16Batch Size (BS)=8, Sequence Length (Seqlen)=1k, block size s=162026.020.078
CIS-16Batch Size (BS)=16, Sequence Length (Seqlen)=1k, block size s=162026.020.081
CIS-16Batch Size (BS)=8, Sequence Length (Seqlen)=1k, block size s=162026.020.083
H2OBatch Size (BS)=8, Sequence Length (Seqlen)=1k2026.020.088
H2OBatch Size (BS)=16, Sequence Length (Seqlen)=1k2026.020.089
CPE-8Batch Size (BS)=16, Sequence Length (Seqlen)=1k, block size s=82026.020.089
HShare-1Batch Size (BS)=8, Sequence Length (Seqlen)=1k, ratio=1/2-1/2-1/22026.020.09
CPE-8Batch Size (BS)=8, Sequence Length (Seqlen)=1k, block size s=82026.020.092
H2OBatch Size (BS)=8, Sequence Length (Seqlen)=2k2026.020.093
HShare-1Batch Size (BS)=16, Sequence Length (Seqlen)=1k, ratio=1/2-1/2-1/22026.020.093
CPE-16Batch Size (BS)=8, Sequence Length (Seqlen)=2k, block size s=162026.020.095
CIS-16Batch Size (BS)=8, Sequence Length (Seqlen)=2k, block size s=162026.020.106
H2OBatch Size (BS)=16, Sequence Length (Seqlen)=2k2026.020.11
CIS-8Batch Size (BS)=16, Sequence Length (Seqlen)=1k, block size s=82026.020.113
CIS-8Batch Size (BS)=8, Sequence Length (Seqlen)=1k, block size s=82026.020.116
HShare-1Batch Size (BS)=8, Sequence Length (Seqlen)=2k, ratio=1/2-1/2-1/22026.020.12
HShare-0Batch Size (BS)=16, Sequence Length (Seqlen)=1k, ratio=3/4-3/4-1/22026.020.12
HShare-0Batch Size (BS)=8, Sequence Length (Seqlen)=1k, ratio=3/4-3/4-1/22026.020.124
DSBatch Size (BS)=8, Sequence Length (Seqlen)=1k2026.020.141
CPE-8Batch Size (BS)=8, Sequence Length (Seqlen)=2k, block size s=82026.020.144
CIS-8Batch Size (BS)=8, Sequence Length (Seqlen)=2k, block size s=82026.020.147
HShare-0Batch Size (BS)=8, Sequence Length (Seqlen)=2k, ratio=3/4-3/4-1/22026.020.16
CPE-16Batch Size (BS)=16, Sequence Length (Seqlen)=2k, block size s=162026.020.165
CIS-16Batch Size (BS)=16, Sequence Length (Seqlen)=2k, block size s=162026.020.178
HShare-1Batch Size (BS)=16, Sequence Length (Seqlen)=2k, ratio=1/2-1/2-1/22026.020.19
CPE-8Batch Size (BS)=16, Sequence Length (Seqlen)=2k, block size s=82026.020.196
QuestBatch Size (BS)=8, Sequence Length (Seqlen)=1k2026.020.2
CIS-8Batch Size (BS)=16, Sequence Length (Seqlen)=2k, block size s=82026.020.214
FlashBatch Size (BS)=8, Sequence Length (Seqlen)=1k2026.020.23
HShare-0Batch Size (BS)=16, Sequence Length (Seqlen)=2k, ratio=3/4-3/4-1/22026.020.23
DSBatch Size (BS)=16, Sequence Length (Seqlen)=1k2026.020.233
DSBatch Size (BS)=8, Sequence Length (Seqlen)=2k2026.020.241
QuestBatch Size (BS)=16, Sequence Length (Seqlen)=1k2026.020.28
DSBatch Size (BS)=16, Sequence Length (Seqlen)=2k2026.020.422
FlashBatch Size (BS)=16, Sequence Length (Seqlen)=1k2026.020.44
QuestBatch Size (BS)=8, Sequence Length (Seqlen)=2k2026.020.46
CPE-16Batch Size (BS)=8, Sequence Length (Seqlen)=4k, block size s=162026.020.469
H2OBatch Size (BS)=8, Sequence Length (Seqlen)=4k2026.020.47
CIS-16Batch Size (BS)=8, Sequence Length (Seqlen)=4k, block size s=162026.020.501
CPE-8Batch Size (BS)=8, Sequence Length (Seqlen)=4k, block size s=82026.020.506
HShare-1Batch Size (BS)=8, Sequence Length (Seqlen)=4k, ratio=1/2-1/2-1/22026.020.53
CIS-8Batch Size (BS)=8, Sequence Length (Seqlen)=4k, block size s=82026.020.534
HShare-0Batch Size (BS)=8, Sequence Length (Seqlen)=4k, ratio=3/4-3/4-1/22026.020.57
DSBatch Size (BS)=8, Sequence Length (Seqlen)=4k2026.020.733
QuestBatch Size (BS)=16, Sequence Length (Seqlen)=2k2026.020.77
CPE-16Batch Size (BS)=16, Sequence Length (Seqlen)=4k, block size s=162026.020.821
FlashBatch Size (BS)=8, Sequence Length (Seqlen)=2k2026.020.83
QuestBatch Size (BS)=8, Sequence Length (Seqlen)=4k2026.020.85
H2OBatch Size (BS)=16, Sequence Length (Seqlen)=4k2026.020.85
CIS-16Batch Size (BS)=16, Sequence Length (Seqlen)=4k, block size s=162026.020.869
CPE-8Batch Size (BS)=16, Sequence Length (Seqlen)=4k, block size s=82026.020.904
CIS-8Batch Size (BS)=16, Sequence Length (Seqlen)=4k, block size s=82026.020.953
HShare-1Batch Size (BS)=16, Sequence Length (Seqlen)=4k, ratio=1/2-1/2-1/22026.020.99
HShare-0Batch Size (BS)=16, Sequence Length (Seqlen)=4k, ratio=3/4-3/4-1/22026.021.041
DSBatch Size (BS)=16, Sequence Length (Seqlen)=4k2026.021.35
FlashBatch Size (BS)=8, Sequence Length (Seqlen)=4k2026.021.63
FlashBatch Size (BS)=16, Sequence Length (Seqlen)=2k2026.021.63
QuestBatch Size (BS)=16, Sequence Length (Seqlen)=4k2026.022.21
FlashBatch Size (BS)=16, Sequence Length (Seqlen)=4k2026.023.23