Attention Operator Throughput on Qwen2.5 72B (64 Q-heads/8 KV-heads/128 Head-dimension)
222.5Attention Throughput (TFLOPS)flash-attn v2
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| flash-attn v2Sequence Length=16k, Hardware=A100 GPU, Head Dimension=1282025.06 | 222.5 | — | — | — | — | |
| flash-attn v2Sequence Length=8k, Hardware=A100 GPU, Head Dimension=1282025.06 | 216 | — | — | — | — | |
| cuDNNSequence Length=16k, Hardware=A100 GPU, Head Dimension=1282025.06 | 207.9 | — | — | — | — | |
| QiMeng-AttentionSequence Length=16k, Hardware=A100 GPU, Head Dimension=1282025.06 | 205.1 | — | — | — | — | |
| cuDNNSequence Length=8k, Hardware=A100 GPU, Head Dimension=1282025.06 | 202.1 | — | — | — | — | |
| flash-attn v2Sequence Length=4k, Hardware=A100 GPU, Head Dimension=1282025.06 | 201.8 | — | — | — | — | |
| QiMeng-AttentionSequence Length=8k, Hardware=A100 GPU, Head Dimension=1282025.06 | 201.8 | — | — | — | — | |
| QiMeng-AttentionSequence Length=4k, Hardware=A100 GPU, Head Dimension=1282025.06 | 192.9 | — | — | — | — | |
| cuDNNSequence Length=4k, Hardware=A100 GPU, Head Dimension=1282025.06 | 191.8 | — | — | — | — | |
| QiMeng-AttentionSequence Length=2k, Hardware=A100 GPU, Head Dimension=1282025.06 | 186.4 | — | — | — | — | |
| flash-attn v2Sequence Length=2k, Hardware=A100 GPU, Head Dimension=1282025.06 | 181.5 | — | — | — | — | |
| cuDNNSequence Length=2k, Hardware=A100 GPU, Head Dimension=1282025.06 | 176.1 | — | — | — | — | |
| FlexAttentionSequence Length=16k, Hardware=A100 GPU, Head Dimension=1282025.06 | 172.4 | — | — | — | — | |
| FlexAttentionSequence Length=8k, Hardware=A100 GPU, Head Dimension=1282025.06 | 168.1 | — | — | — | — | |
| QiMeng-AttentionSequence Length=1k, Hardware=A100 GPU, Head Dimension=1282025.06 | 165.1 | — | — | — | — | |
| flash-attn v2Sequence Length=1k, Hardware=A100 GPU, Head Dimension=1282025.06 | 157.8 | — | — | — | — | |
| FlexAttentionSequence Length=4k, Hardware=A100 GPU, Head Dimension=1282025.06 | 154.7 | — | — | — | — | |
| cuDNNSequence Length=1k, Hardware=A100 GPU, Head Dimension=1282025.06 | 148.3 | — | — | — | — | |
| QiMeng-AttentionSequence Length=512, Hardware=A100 GPU, Head Dimension=1282025.06 | 140.2 | — | — | — | — | |
| FlexAttentionSequence Length=2k, Hardware=A100 GPU, Head Dimension=1282025.06 | 136.7 | — | — | — | — | |
| flash-attn v2Sequence Length=512, Hardware=A100 GPU, Head Dimension=1282025.06 | 127.1 | — | — | — | — | |
| FlexAttentionSequence Length=1k, Hardware=A100 GPU, Head Dimension=1282025.06 | 117.4 | — | — | — | — | |
| cuDNNSequence Length=512, Hardware=A100 GPU, Head Dimension=1282025.06 | 116.3 | — | — | — | — | |
| FlexAttentionSequence Length=512, Hardware=A100 GPU, Head Dimension=1282025.06 | 86.1 | — | — | — | — | |
| DeepSeek-V3Sequence Length=8k, Hardware=A100 GPU, Head Dimension=1282025.06 | 11.2 | — | — | — | — | |
| DeepSeek-V3Sequence Length=1k, Hardware=A100 GPU, Head Dimension=1282025.06 | 11.1 | — | — | — | — | |
| DeepSeek-V3Sequence Length=512, Hardware=A100 GPU, Head Dimension=1282025.06 | 10.9 | — | — | — | — | |
| DeepSeek-V3Sequence Length=4k, Hardware=A100 GPU, Head Dimension=1282025.06 | 10.2 | — | — | — | — | |
| DeepSeek-V3Sequence Length=2k, Hardware=A100 GPU, Head Dimension=1282025.06 | 8.8 | — | — | — | — | |
| CuBridgeHardware=NVIDIA H100, Heads (q, k)=64, 82026.05 | — | 304.35 | 486.19 | 599.14 | 615.26 | |
| CuBridgeHardware=NVIDIA H1002026.05 | — | 521.33 | 632.6 | 622.52 | 637.52 | |
| FlexAttentionHardware=NVIDIA H100, Heads (q, k)=64, 82026.05 | — | 248.79 | 333.7 | 382.13 | 402.38 | |
| Qimeng AttnHardware=NVIDIA H100, Heads (q, k)=64, 82026.05 | — | 113.24 | 163.79 | 207.47 | 232.61 | |
| Qimeng AttnHardware=NVIDIA H1002026.05 | — | 253.49 | 346.74 | 415 | 443.19 | |
| TorchHardware=NVIDIA H100, Heads (q, k)=64, 82026.05 | — | 28.4 | 19.99 | 25.67 | 29.77 | |
| TorchHardware=NVIDIA H1002026.05 | — | 72.52 | 82.3 | 88.87 | 92.94 |