Multi-Head Attention Performance on A100 GPU (hd=64, sl=1024)
175.6TFLOPSLLM-TL
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| LLM-TLImplementation Library=CuTe, Hardware=A100 GPU, Head Dimension=64, Sequence Length=10242025.06 | 175.6 | 10 | |
| Human ExpertImplementation Library=CuTe, Hardware=A100 GPU, Head Dimension=64, Sequence Length=10242025.06 | 162.7 | — |