Inference Efficiency on LLaMA 8B 32K context length 3.1
1,115Theoretical Compute (TFLOPs)SpecKV
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| SpecKVCache Budget (C)=1282026.03 | 1,115 | 106 | 2,156 | 402.8 | 2,263 | 503 | |
| LAQCache Budget (C)=1282026.03 | 930 | 451 | 1,993 | 239.26 | 2,314 | 554 | |
| LOOKAHEADKVCache Budget (C)=1282026.03 | 929 | 13 | 1,755 | 1.74 | 1,798 | 38 | |
| Forward Pass OnlyCache Budget (C)=1282026.03 | 928 | 13 | 1,754 | — | 1,760 | — | |
| SnapKVCache Budget (C)=1282026.03 | 928 | 13 | 1,754 | 0.01 | 1,838 | 78 |