Decoding Latency on Llama-3.1-8B 64k sequence length v1
0.05Decoding Latency (s)Full Cache
Evaluation Results
| Method | Links | |
|---|---|---|
| Full CacheHardware=NVIDIA A100 (80GB, PCIe 4.0 link)2025.05 | 0.05 | |
| TailorKVHardware=NVIDIA A100 (80GB, PCIe 4.0 link)2025.05 | 0.054 | |
| TailorKVHardware=NVIDIA RTX 3090 (24GB, PCIe 1.0 link)2025.05 | 0.074 | |
| PQCacheHardware=NVIDIA A100 (80GB, PCIe 4.0 link)2025.05 | 0.108 | |
| OffloadCacheHardware=NVIDIA A100 (80GB, PCIe 4.0 link)2025.05 | 0.435 |