Token Throughput on LLaMA 8B 3.1 (4K-128K Context)
150.8Token Throughput (4K Context) (tokens/sec)ShadowKV
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| ShadowKVBatch size=82026.03 | 150.8 | 136.7 | 105.2 | 67.5 | 34.9 | |
| TTKVBatch size=82026.03 | 148.7 | 142.1 | 128 | 95.3 | 65.7 | |
| KVQuant (3-bit)Batch size=82026.03 | 145.1 | 130.5 | 98.6 | 61 | 30.8 | |
| KIVI (2-bit)Batch size=82026.03 | 142.3 | 128 | 95 | 58.7 | 29.4 | |
| FP16Batch size=82026.03 | 135.5 | 120.4 | 89.2 | — | — |