Multi-Head Attention (MHA) on NVIDIA A100 GPU
207.2TFLOPSDeepSeek-V3 + Ours
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| DeepSeek-V3 + OursHead Dimension=128, Sequence Length=16k, Causal Mask=false2025.06 | 207.2 | 3.73 | |
| flash-attn v2Head Dimension=128, Sequence Length=16k, Causal Mask=true2025.06 | 195.1 | — | |
| DeepSeek-V3 + OursHead Dimension=64, Sequence Length=16k, Causal Mask=true2025.06 | 184.3 | 34.94 | |
| cuDNNHead Dimension=64, Sequence Length=512, Causal Mask=true2025.06 | 95.3 | — | |
| DeepSeek-V3Head Dimension=64, Sequence Length=16k, Causal Mask=true2025.06 | 7.7 | — |