Multi-Query Attention on T4 GPU Synthetic Performance Benchmark
19.46Performance (TFLOPS)DeepSeek-V3 + Ours
Evaluation results
| Method | Links | |
|---|---|---|
| DeepSeek-V3 + OursHead Dimension=128, Sequence Length=16k2025.06 | 19.46 | |
| flash-attn v1Head Dimension=64, Sequence Length=1k2025.06 | 15.32 |