Inference Efficiency on 32k Context Length Llama-3-8B (test)
4.12Time To First Token (s)Full Model
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| Full ModelModel=Llama-3-8B, Context Length=32k, KV cache budget=100%2026.03 | 4.12 | 0.081 | 24.27 | |
| StreamingLLMModel=Llama-3-8B, Context Length=32k, KV cache budget=20%2026.03 | 4.12 | 0.032 | 16.12 | |
| H2OModel=Llama-3-8B, Context Length=32k, KV cache budget=20%2026.03 | 4.12 | 0.033 | 16.58 | |
| SnapKVModel=Llama-3-8B, Context Length=32k, KV cache budget=20%2026.03 | 4.13 | 0.032 | 16.37 | |
| SemantiCacheModel=Llama-3-8B, Context Length=32k, KV cache budget=20%2026.03 | 4.25 | 0.031 | 15.94 | |
| CaMModel=Llama-3-8B, Context Length=32k, KV cache budget=20%2026.03 | 4.28 | 0.039 | 17.03 | |
| D2OModel=Llama-3-8B, Context Length=32k, KV cache budget=20%2026.03 | 4.29 | 0.038 | 16.91 |