LLM Inference Efficiency on Synthetic LLM Workload (4K Input/4K Output)
160.32Latency (s)ChunkKV
Evaluation results
| Method | Links | ||
|---|---|---|---|
| ChunkKVInput Sequence Length=4096, Output Sequence Length=4096, Batch size=1, Hardware server=A40 server2025.02 | 160.32 | 41.3 | |
| ShotKVInput Sequence Length=4096, Output Sequence Length=4096, Batch size=1, Hardware server=A40 server2025.02 | 162.85 | 41.12 | |
| SnapKVInput Sequence Length=4096, Output Sequence Length=4096, Batch size=1, Hardware server=A40 server2025.02 | 163.45 | 40.51 | |
| FullKVInput Sequence Length=4096, Output Sequence Length=4096, Batch size=1, Hardware server=A40 server2025.02 | 175.5 | 37.73 |