Paged Decode Attention Serving on Bucketed B1 trace
1.576GPU RatioPersistentKV
Evaluation Results
| Method | Links | ||||
|---|---|---|---|---|---|
| PersistentKVSelected route=PersistentKV bucket, split 32, Page size=16, Hq=32, Hkv=8, d=128, Precision=FP162026.06 | 1.576 | 1.399 | 6.104 | 4.398 |
Loading the SOTA2 catalog…