End-to-end Attention Latency on 120K Context
62.9Attention Latency (120K Context, µs)MAC-Attn.
Evaluation results
| Method | Links | |
|---|---|---|
| MAC-Attn.KV %=12026.03 | 62.9 | |
| MAC-Attn.KV %=52026.03 | 64 | |
| MAC-Attn.KV %=102026.03 | 78.1 | |
| MAC-Attn.KV %=202026.03 | 103.8 | |
| MultipoleKV %=12026.03 | 192.4 | |
| MultipoleKV %=52026.03 | 210.8 | |
| Full Attn.KV %=12026.03 | 234.2 | |
| Full Attn.KV %=52026.03 | 234.2 | |
| Full Attn.KV %=102026.03 | 234.2 | |
| Full Attn.KV %=202026.03 | 234.2 | |
| MultipoleKV %=102026.03 | 265.4 | |
| MultipoleKV %=202026.03 | 324.6 | |
| QuestKV %=12026.03 | 581.2 | |
| QuestKV %=52026.03 | 594.7 | |
| QuestKV %=102026.03 | 608.5 | |
| QuestKV %=202026.03 | 640.5 | |
| RocketKVKV %=12026.03 | 822.8 | |
| RocketKVKV %=52026.03 | 844.7 | |
| RocketKVKV %=102026.03 | 1,042.5 | |
| RocketKVKV %=202026.03 | 1,855.6 |