Multi-Query Attention (MQA) Performance on NVIDIA A100 GPU
211.4TFLOPScuDNN
Evaluation results
| Method | Links | ||
|---|---|---|---|
| cuDNNHead Dimension=128, Sequence Length=16k, Causal Mask=false2025.06 | 211.4 | — | |
| DeepSeek-V3 + OursHead Dimension=64, Sequence Length=512, Causal Mask=false2025.06 | 156.5 | 5.36 |