Attention Operator Throughput on Llama2 7B (32 Q-heads/32 KV-heads/128 Head-dimension)
207.3Attention TFLOPSflash-attn v2
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| flash-attn v2Sequence Length=16k, Hardware=A100 GPU, Head Dimension=128, Attention Mask=causal2025.06 | 207.3 | — | — | — | — | |
| QiMeng-AttentionSequence Length=16k, Hardware=A100 GPU, Head Dimension=128, Attention Mask=causal2025.06 | 202.7 | — | — | — | — | |
| flash-attn v2Sequence Length=8k, Hardware=A100 GPU, Head Dimension=128, Attention Mask=causal2025.06 | 201.5 | — | — | — | — | |
| cuDNNSequence Length=16k, Hardware=A100 GPU, Head Dimension=128, Attention Mask=causal2025.06 | 201.2 | — | — | — | — | |
| QiMeng-AttentionSequence Length=8k, Hardware=A100 GPU, Head Dimension=128, Attention Mask=causal2025.06 | 198.3 | — | — | — | — | |
| cuDNNSequence Length=8k, Hardware=A100 GPU, Head Dimension=128, Attention Mask=causal2025.06 | 197.2 | — | — | — | — | |
| flash-attn v2Sequence Length=4k, Hardware=A100 GPU, Head Dimension=128, Attention Mask=causal2025.06 | 188.3 | — | — | — | — | |
| QiMeng-AttentionSequence Length=4k, Hardware=A100 GPU, Head Dimension=128, Attention Mask=causal2025.06 | 186.7 | — | — | — | — | |
| QiMeng-AttentionSequence Length=2k, Hardware=A100 GPU, Head Dimension=128, Attention Mask=causal2025.06 | 180.3 | — | — | — | — | |
| cuDNNSequence Length=4k, Hardware=A100 GPU, Head Dimension=128, Attention Mask=causal2025.06 | 176.8 | — | — | — | — | |
| flash-attn v2Sequence Length=2k, Hardware=A100 GPU, Head Dimension=128, Attention Mask=causal2025.06 | 173.4 | — | — | — | — | |
| FlexAttentionSequence Length=16k, Hardware=A100 GPU, Head Dimension=128, Attention Mask=causal2025.06 | 167.1 | — | — | — | — | |
| cuDNNSequence Length=2k, Hardware=A100 GPU, Head Dimension=128, Attention Mask=causal2025.06 | 164.1 | — | — | — | — | |
| QiMeng-AttentionSequence Length=1k, Hardware=A100 GPU, Head Dimension=128, Attention Mask=causal2025.06 | 160.6 | — | — | — | — | |
| FlexAttentionSequence Length=8k, Hardware=A100 GPU, Head Dimension=128, Attention Mask=causal2025.06 | 158.4 | — | — | — | — | |
| flash-attn v2Sequence Length=1k, Hardware=A100 GPU, Head Dimension=128, Attention Mask=causal2025.06 | 152.5 | — | — | — | — | |
| FlexAttentionSequence Length=4k, Hardware=A100 GPU, Head Dimension=128, Attention Mask=causal2025.06 | 150.2 | — | — | — | — | |
| CuBridgeImplementation=CuBridge2026.05 | 145.69 | — | — | — | — | |
| cuDNNSequence Length=1k, Hardware=A100 GPU, Head Dimension=128, Attention Mask=causal2025.06 | 142.6 | — | — | — | — | |
| QiMeng-AttentionSequence Length=512, Hardware=A100 GPU, Head Dimension=128, Attention Mask=causal2025.06 | 137.1 | — | — | — | — | |
| FlexAttentionImplementation=PyTorch FlexAttention2026.05 | 131.67 | — | — | — | — | |
| FlexAttentionSequence Length=2k, Hardware=A100 GPU, Head Dimension=128, Attention Mask=causal2025.06 | 128.6 | — | — | — | — | |
| flash-attn v2Sequence Length=512, Hardware=A100 GPU, Head Dimension=128, Attention Mask=causal2025.06 | 122.5 | — | — | — | — | |
| CuBridgeImplementation=CuBridge2026.05 | 116.69 | — | — | — | — | |
| cuDNNSequence Length=512, Hardware=A100 GPU, Head Dimension=128, Attention Mask=causal2025.06 | 112.4 | — | — | — | — | |
| FlexAttentionImplementation=PyTorch FlexAttention2026.05 | 110.44 | — | — | — | — | |
| FlexAttentionSequence Length=1k, Hardware=A100 GPU, Head Dimension=128, Attention Mask=causal2025.06 | 108.6 | — | — | — | — | |
| Qimeng AttnImplementation=Qimeng Attention2026.05 | 105.55 | — | — | — | — | |
| CuBridgeImplementation=CuBridge2026.05 | 92.5 | — | — | — | — | |
| FlexAttentionSequence Length=512, Hardware=A100 GPU, Head Dimension=128, Attention Mask=causal2025.06 | 82.8 | — | — | — | — | |
| Qimeng AttnImplementation=Qimeng Attention2026.05 | 79.37 | — | — | — | — | |
| FlexAttentionImplementation=PyTorch FlexAttention2026.05 | 77.09 | — | — | — | — | |
| Qimeng AttnImplementation=Qimeng Attention2026.05 | 44.46 | — | — | — | — | |
| DeepSeek-V3Sequence Length=1k, Hardware=A100 GPU, Head Dimension=128, Attention Mask=causal2025.06 | 15.1 | — | — | — | — | |
| DeepSeek-V3Sequence Length=16k, Hardware=A100 GPU, Head Dimension=128, Attention Mask=causal2025.06 | 15.1 | — | — | — | — | |
| DeepSeek-V3Sequence Length=512, Hardware=A100 GPU, Head Dimension=128, Attention Mask=causal2025.06 | 14.6 | — | — | — | — | |
| DeepSeek-V3Sequence Length=8k, Hardware=A100 GPU, Head Dimension=128, Attention Mask=causal2025.06 | 14.6 | — | — | — | — | |
| DeepSeek-V3Sequence Length=4k, Hardware=A100 GPU, Head Dimension=128, Attention Mask=causal2025.06 | 13.3 | — | — | — | — | |
| TorchImplementation=Standard PyTorch2026.05 | 13.12 | — | — | — | — | |
| TorchImplementation=Standard PyTorch2026.05 | 12.36 | — | — | — | — | |
| TorchImplementation=Standard PyTorch2026.05 | 10.99 | — | — | — | — | |
| DeepSeek-V3Sequence Length=2k, Hardware=A100 GPU, Head Dimension=128, Attention Mask=causal2025.06 | 10.9 | — | — | — | — | |
| CuBridgeHardware=NVIDIA H100, Heads (q, k)=32, 322026.05 | — | 304.35 | 414.05 | 577.03 | 551.73 | |
| CuBridgeHardware=NVIDIA H1002026.05 | — | 137.8 | 176.88 | 182.44 | 151.9 | |
| FlexAttentionHardware=NVIDIA H100, Heads (q, k)=32, 322026.05 | — | 211.17 | 319.03 | 374.71 | 404.09 | |
| FlexAttentionHardware=NVIDIA H1002026.05 | — | 109.37 | 111.62 | 113.39 | 115.7 | |
| Qimeng AttnHardware=NVIDIA H100, Heads (q, k)=32, 322026.05 | — | 81.07 | 126.08 | 175.6 | 215.75 | |
| Qimeng AttnHardware=NVIDIA H1002026.05 | — | 65.43 | 53.43 | 38.77 | 24.62 | |
| TorchHardware=NVIDIA H100, Heads (q, k)=32, 322026.05 | — | 27.22 | 19.72 | 25.48 | 29.61 | |
| TorchHardware=NVIDIA H1002026.05 | — | 23.65 | 9.25 | 6 | 3.4 |