Summarization on GovReport 16K (Latency, Memory, ROUGE)
11895th Percentile Latency (ms)TTKV
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| TTKVModel=Llama-3.1-8B2026.03 | 118 | 0.62 | 48.4 | |
| TTKVModel=DeepSeek-R1-14B2026.03 | 175 | 0.85 | 50 | |
| KIVIModel=Llama-3.1-8B, Bit-width=2-bit2026.03 | 178 | 1.4 | 47.8 | |
| KVQuantModel=Llama-3.1-8B, Bit-width=3-bit2026.03 | 182 | 1.5 | 48.3 | |
| FP16Model=Llama-3.1-8B2026.03 | 185 | 1.6 | 48.5 | |
| DiffKVModel=Llama-3.1-8B2026.03 | 185 | 1.55 | 48.1 | |
| ShadowKVModel=Llama-3.1-8B2026.03 | 188 | 1.58 | 48.2 | |
| KIVIModel=DeepSeek-R1-14B, Bit-width=2-bit2026.03 | 275 | 2.15 | 48.8 | |
| KVQuantModel=DeepSeek-R1-14B, Bit-width=3-bit2026.03 | 280 | 2.25 | 50 | |
| FP16Model=DeepSeek-R1-14B2026.03 | 285 | 2.4 | 50.2 | |
| DiffKVModel=DeepSeek-R1-14B2026.03 | 285 | 2.3 | 49.2 | |
| ShadowKVModel=DeepSeek-R1-14B2026.03 | 290 | 2.38 | 49.9 |