Inference Efficiency on LLaMA 8B 8K context length 3.1
159Theoretical Compute (TFLOPs)SpecKV
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| SpecKVCache Budget (C)=1282026.03 | 159 | 81 | 337 | 79.53 | 411 | 121 | |
| SpecKV2026.03 | 159 | 81 | 337 | 79.53 | 411 | 120.51 | |
| LOOKAHEADKVCache Budget (C)=1282026.03 | 137 | 13 | 258 | 1.03 | 302 | 11 | |
| LAQCache Budget (C)=1282026.03 | 137 | 445 | 492 | 234.59 | 800 | 509 | |
| LOOKAHEADKV2026.03 | 137 | 13 | 258 | 1.03 | 302 | 10.88 | |
| LAQ2026.03 | 137 | 445 | 492 | 234.59 | 800 | 509.38 | |
| Forward Pass OnlyCache Budget (C)=1282026.03 | 136 | 13 | 257 | — | 291 | — | |
| SnapKVCache Budget (C)=1282026.03 | 136 | 13 | 257 | 0.01 | 311 | 20 | |
| Forward Pass Only2026.03 | 136 | 13 | 257 | — | 291 | — | |
| SnapKV2026.03 | 136 | 13 | 257 | 0.01 | 311 | 20.17 |