LLM Inference on LLaMA-2 70B sequence length 2048
384Max Batch SizeCXL-SpecKV + Comp
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| CXL-SpecKV + CompSystem=CXL-SpecKV + Comp2025.12 | 384 | — | 24 | |
| CPU OffloadSystem=CPU Offload2025.12 | 192 | — | 12 | |
| CXL-SpecKVSystem=CXL-SpecKV2025.12 | 128 | — | 8 | |
| GPU + CompressionSystem=GPU + Compression2025.12 | 48 | — | 3 | |
| GPU-OnlySystem=GPU-Only2025.12 | 16 | — | 1 |