Text Generation Throughput on R1-Llama-8B 32K generation
2.51Memory Footprint (%)ThinKV
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| ThinKVTok. Budget=1024, Evaluation Protocol=Standard2025.10 | 2.51 | 711 | 8,412.2 | 938 | 10,578.5 | |
| R-KV (seq)Tok. Budget=1024, Evaluation Protocol=Standard2025.10 | 5.48 | 268 | 1,450.5 | 350 | 2,425.8 | |
| R-KV (ovl)Tok. Budget=1024, Evaluation Protocol=Standard2025.10 | 5.48 | 268 | 2,320.9 | 350 | 4,311.3 | |
| R-KV (seq)Tok. Budget=1024, Evaluation Protocol=Iso-batch, Iso-compression comparison2025.10 | 5.48 | 256 | 1,769.3 | 256 | 2,489.8 | |
| R-KV (ovl)Tok. Budget=1024, Evaluation Protocol=Iso-batch, Iso-compression comparison2025.10 | 5.48 | 256 | 3,539.3 | 256 | 5,318.7 | |
| ThinKV w/o TBQTok. Budget=1024, Evaluation Protocol=Iso-batch, Iso-compression comparison2025.10 | 5.78 | 256 | 5,298.4 | 256 | 8,079.9 | |
| FullKVEvaluation Protocol=Standard2025.10 | 100 | 13 | 297.5 | 19 | 453.9 |