Throughput Evaluation on vLLM
6,622.4ThroughputThinKV
Evaluation results
| Method | Links | |
|---|---|---|
| ThinKVBatch Size=256, Budget=10242025.10 | 6,622.4 | |
| R-KV (ovl)Batch Size=256, Budget=10242025.10 | 4,883.3 | |
| ThinKVBatch Size=8, Budget=10242025.10 | 346.9 | |
| R-KV (ovl)Batch Size=8, Budget=10242025.10 | 331.9 | |
| FullKVBatch Size=8, Budget=–2025.10 | 228.5 |