List your company
ResearchFormerly WizWand

Decoding Latency on Llama-3.1-8B 16k sequence length v1 (inference)

0.024Decoding latency (s)Full Cache
0.015280.074140.1330.19186May 26, 2025
Updated 8mo ago

Evaluation results

MethodLinks
Full CacheHardware=NVIDIA A100 (80GB, PCIe 4.0 link)2025.050.024
Full CacheHardware=NVIDIA RTX 3090 (24GB, PCIe 1.0 link)2025.050.033
TailorKVHardware=NVIDIA A100 (80GB, PCIe 4.0 link)2025.050.045
TailorKVHardware=NVIDIA RTX 3090 (24GB, PCIe 1.0 link)2025.050.062
PQCacheHardware=NVIDIA A100 (80GB, PCIe 4.0 link)2025.050.104
OffloadCacheHardware=NVIDIA A100 (80GB, PCIe 4.0 link)2025.050.124
PQCacheHardware=NVIDIA RTX 3090 (24GB, PCIe 1.0 link)2025.050.126
OffloadCacheHardware=NVIDIA RTX 3090 (24GB, PCIe 1.0 link)2025.050.242