Masked Multi-Query Attention on T4 GPU Synthetic Performance Benchmark
16.7Performance (TFLOPS)DeepSeek-V3 + Ours
Evaluation Results
| Method | Links | |
|---|---|---|
| DeepSeek-V3 + OursHead Dimension=128, Sequence Length=2k2025.06 | 16.7 | |
| flash-attn v1Head Dimension=64, Sequence Length=4k2025.06 | 13.74 |