Decoding Latency on Llama-2-7B 96k sequence length v1 (inference)
0.115Decoding Latency (s)PQCache
Evaluation Results
| Method | Links | |
|---|---|---|
| PQCacheHardware=NVIDIA A100 (80GB, PCIe 4.0 link)2025.05 | 0.115 | |
| TailorKVHardware=NVIDIA A100 (80GB, PCIe 4.0 link)2025.05 | 0.132 | |
| TailorKVHardware=NVIDIA RTX 3090 (24GB, PCIe 1.0 link)2025.05 | 0.176 | |
| OffloadCacheHardware=NVIDIA A100 (80GB, PCIe 4.0 link)2025.05 | 3.253 |