Inference Performance Evaluation on GPU Context Length Sweep (2K-8K)
116TTFTSpectrumKV
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| SpectrumKVBackbone=Mistral-7B, Sequence length=2K, KV Cache Budget (b)=0.52026.06 | 116 | 58 | 50.1 | 3 | |
| SpectrumKVBackbone=Gemma-2-9B, Sequence length=2K, KV Cache Budget (b)=0.52026.06 | 135 | 50 | 38.9 | 2 | |
| SpectrumKVBackbone=Qwen2.5-7B, Sequence length=2K, KV Cache Budget (b)=0.52026.06 | 154 | 61 | 43.5 | 11 | |
| SpectrumKVBackbone=Mistral-7B, Sequence length=4K, KV Cache Budget (b)=0.52026.06 | 233 | 53 | 49.1 | 4 | |
| Gemma-2-9BSequence length=2K, KV Cache Budget (b)=Full2026.06 | 269 | — | 38.2 | — | |
| SpectrumKVBackbone=Gemma-2-9B, Sequence length=4K, KV Cache Budget (b)=0.52026.06 | 270 | 62 | 38.2 | 33 | |
| Mistral-7BSequence length=2K, KV Cache Budget (b)=Full2026.06 | 274 | — | 51.9 | — | |
| SpectrumKVBackbone=Qwen2.5-7B, Sequence length=4K, KV Cache Budget (b)=0.52026.06 | 296 | 53 | 43.2 | 2 | |
| Qwen2.5-7BSequence length=2K, KV Cache Budget (b)=Full2026.06 | 391 | — | 49.2 | — | |
| Mistral-7BSequence length=4K, KV Cache Budget (b)=Full2026.06 | 497 | — | 47.1 | — | |
| SpectrumKVBackbone=Mistral-7B, Sequence length=8K, KV Cache Budget (b)=0.52026.06 | 502 | 59 | 47 | 7 | |
| Qwen2.5-7BSequence length=4K, KV Cache Budget (b)=Full2026.06 | 630 | — | 42.4 | — | |
| Gemma-2-9BSequence length=4K, KV Cache Budget (b)=Full2026.06 | 709 | — | 28.8 | — | |
| SpectrumKVBackbone=Gemma-2-9B, Sequence length=8K, KV Cache Budget (b)=0.52026.06 | 713 | 57 | 28.8 | 5 | |
| Mistral-7BSequence length=8K, KV Cache Budget (b)=Full2026.06 | 1,211 | — | 44.1 | — | |
| Gemma-2-9BSequence length=8K, KV Cache Budget (b)=Full2026.06 | 1,664 | — | 27.3 | — |