LLM Inference Latency on 256 tokens generation (test)
26I/O Latency (ms)All (Predictor, Windowing, Bundling)
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| All (Predictor, Windowing, Bundling)Model=OPT 6.7B, Backend=Metal M22023.12 | 26 | 8 | 271 | 305 | |
| All (Predictor, Windowing, Bundling)Model=OPT 6.7B, Backend=GPU2023.12 | 30 | 34 | 20 | 84 | |
| Speculative DecodingModel=OPT 6.7B, Backend=GPU2023.12 | 38.5 | 9.5 | 12 | 60 | |
| All (Predictor, Windowing, Bundling)Model=OPT 6.7B, Backend=Metal M12023.12 | 92 | 35 | 438 | 565 | |
| All (Predictor, Windowing, Bundling)Model=OPT 6.7B, Backend=CPU2023.12 | 105 | 58 | 506 | 669 | |
| All (Predictor, Windowing, Bundling)Model=Falcon 7B, Backend=CPU2023.12 | 161 | 92 | 453 | 706 | |
| All (Predictor, Windowing, Bundling)Model=Phi-2 2.7B, Backend=CPU2023.12 | 211 | 76 | 259 | 546 | |
| All (Predictor, Windowing, Bundling)Model=Llama 2 7B, Backend=CPU2023.12 | 279 | 152 | 563 | 994 | |
| All (Predictor, Windowing, Bundling)Model=Persimmon 8B, Backend=CPU2023.12 | 283 | 98 | 660 | 1,041 | |
| HybridModel=Phi-2 2.7B, Backend=CPU2023.12 | 309 | 0 | 402 | 711 | |
| NaiveModel=Phi-2 2.7B, Backend=CPU2023.12 | 885 | 0 | 402 | 1,287 | |
| HybridModel=Llama 2 7B, Backend=CPU2023.12 | 974 | 0 | 929 | 1,903 | |
| HybridModel=Falcon 7B, Backend=CPU2023.12 | 1,147 | 0 | 800 | 1,947 | |
| HybridModel=Persimmon 8B, Backend=CPU2023.12 | 1,311 | 0 | 1,184 | 2,495 | |
| NaiveModel=OPT 6.7B, Backend=Metal M22023.12 | 2,145 | 0 | 125 | 2,270 | |
| NaiveModel=Llama 2 7B, Backend=CPU2023.12 | 2,166 | 0 | 929 | 3,095 | |
| NaiveModel=OPT 6.7B, Backend=CPU2023.12 | 2,196 | 0 | 986 | 3,182 | |
| NaiveModel=OPT 6.7B, Backend=Metal M12023.12 | 2,196 | 0 | 193 | 2,389 | |
| NaiveModel=OPT 6.7B, Backend=GPU2023.12 | 2,196 | 0 | 22 | 2,218 | |
| NaiveModel=Falcon 7B, Backend=CPU2023.12 | 2,295 | 0 | 800 | 3,095 | |
| NaiveModel=Persimmon 8B, Backend=CPU2023.12 | 2,622 | 0 | 1,184 | 3,806 |