Masked Multi-Head Attention on T4 GPU Synthetic Performance Benchmark
19.07Performance (TFLOPS)DeepSeek-V3 + Ours
Evaluation results
| Method | Links | |
|---|---|---|
| DeepSeek-V3 + OursHead Dimension=128, Sequence Length=16k2025.06 | 19.07 | |
| Flex AttentionHead Dimension=64, Sequence Length=1k2025.06 | 13.45 | |
| flash-attn v1Head Dimension=64, Sequence Length=2k2025.06 | 12.81 | |
| cuDNNHead Dimension=64, Sequence Length=5122025.06 | 8.11 | |
| DeepSeek-V3Head Dimension=128, Sequence Length=4k2025.06 | 2.26 |