CPU Inference Performance Evaluation on Phi-mini-MoE-Instruct
4.1Memory Usage (GB)Llama.cpp (CPU)
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Llama.cpp (CPU)Bit Width=A8W4, Hardware=Intel(R) Xeon(R) w7-3445 CPU, Threads=202026.04 | 4.1 | 15 | |
| CodeQuant (CPU)Bit Width=A8W4, Hardware=Intel(R) Xeon(R) w7-3445 CPU, Threads=202026.04 | 4.1 | 13.3 | |
| Llama.cpp (CPU)Bit Width=BF16, Hardware=Intel(R) Xeon(R) w7-3445 CPU, Threads=202026.04 | 14.3 | 40.1 |