LLM Serving on LLaMA-2 70B chatbot workload
45.2TTFT (ms)GPU-Only
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| GPU-OnlyBatch size=322025.12 | 45.2 | 18.3 | 19.4 | 21.2 | |
| CXL-NoSpecBatch size=322025.12 | 46.8 | 23.5 | 27.3 | 31.6 | |
| CXL-SpecKVBatch size=322025.12 | 47.1 | 19.8 | 21.7 | 23.8 | |
| CPU OffloadBatch size=322025.12 | 52.7 | 28.6 | 35.4 | 42.8 |