Chatbot Workload Efficiency on LLaMA 2 70B
315GPU Power (W)GPU-Only
Evaluation Results
| Method | Links | ||||
|---|---|---|---|---|---|
| GPU-Onlybatch size=162025.12 | 315 | 0 | 315 | 0.647 | |
| CXL-SpecKVbatch size=642025.12 | 342 | 184 | 526 | 0.34 |
Loading the SOTA2 catalog…
| Method | Links | ||||
|---|---|---|---|---|---|
| GPU-Onlybatch size=162025.12 | 315 | 0 | 315 | 0.647 | |
| CXL-SpecKVbatch size=642025.12 | 342 | 184 | 526 | 0.34 |