Multi-Head Attention (MHA) Performance on NVIDIA RTX8000 GPU
49.9TFLOPSDeepSeek-V3 + Ours
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| DeepSeek-V3 + OursHead Dimension=64, Sequence Length=16k, Causal Mask=true2025.06 | 49.9 | — | |
| flash-attn v1Head Dimension=64, Sequence Length=512, Causal Mask=true2025.06 | 18.1 | — |