CPU Inference Performance Evaluation on DeepSeek Lite V2
8.8Memory Usage (GB)Llama.cpp (CPU)
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Llama.cpp (CPU)Bit Width=A8W4, Hardware=Intel(R) Xeon(R) w7-3445 CPU, Threads=202026.04 | 8.8 | 17.1 | |
| CodeQuant (CPU)Bit Width=A8W4, Hardware=Intel(R) Xeon(R) w7-3445 CPU, Threads=202026.04 | 8.9 | 14.2 | |
| Llama.cpp (CPU)Bit Width=BF16, Hardware=Intel(R) Xeon(R) w7-3445 CPU, Threads=202026.04 | 29.3 | 50 |