Paged Decode Attention Serving on Zipf B8 trace
1.157GPU RatioPersistentKV
Evaluation Results
| Method | Links | ||||
|---|---|---|---|---|---|
| PersistentKVSelected route=PersistentKV workqueue, split 28, Page size=16, Hq=32, Hkv=8, d=128, Precision=FP162026.06 | 1.157 | 1.063 | 0 | 0 |
Loading the SOTA2 catalog…