LLM Inference Acceleration on MBPP
6.26SpeedupFLy
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| FLyModel=Llama-3.1-Instruct-405B, Training-based=false, Temperature=02025.11 | 6.26 | 18.56 | |
| FLyModel=Llama-3.1-Instruct-405B, Training-based=false, Temperature=12025.11 | 5.97 | 17.8 | |
| EAGLE-3Model=Meta-Llama-3.3-Instruct-70B, Training-based=true, Temperature=02025.11 | 3.8 | 5.47 | |
| EAGLE-3Model=Meta-Llama-3.3-Instruct-70B, Training-based=true, Temperature=12025.11 | 3.58 | 5.15 | |
| SpSModel=Llama-3.1-Instruct-405B, Training-based=false, Temperature=02025.11 | 3.03 | 12.67 | |
| FLyModel=Llama-3.1-Instruct-70B, Training-based=false, Temperature=02025.11 | 2.79 | 12.84 | |
| FLyModel=Meta-Llama-3.3-Instruct-70B, Training-based=false, Temperature=12025.11 | 2.69 | 12.33 | |
| EAGLE-2Model=Meta-Llama-3-Instruct-70B, Training-based=true, Temperature=02025.11 | 2.61 | 3.9 | |
| FLyModel=Meta-Llama-3-Instruct-70B, Training-based=false, Temperature=12025.11 | 2.59 | 11.37 | |
| FLyModel=Llama-3.1-Instruct-70B, Training-based=false, Temperature=12025.11 | 2.59 | 12.57 | |
| SpSModel=Llama-3.1-Instruct-405B, Training-based=false, Temperature=12025.11 | 2.55 | 10.67 | |
| FLyModel=Meta-Llama-3-Instruct-70B, Training-based=false, Temperature=02025.11 | 2.53 | 11.21 | |
| FLyModel=Meta-Llama-3.3-Instruct-70B, Training-based=false, Temperature=02025.11 | 2.47 | 11.76 | |
| EAGLE-2Model=Meta-Llama-3-Instruct-70B, Training-based=true, Temperature=12025.11 | 2.43 | 3.63 | |
| RESTModel=Llama-3.1-Instruct-405B, Training-based=false, Temperature=02025.11 | 2.36 | 2.46 | |
| TokenRecyclingModel=Llama-3.1-Instruct-70B, Training-based=false, Temperature=12025.11 | 2.15 | 3.01 | |
| TokenRecyclingModel=Llama-3.1-Instruct-70B, Training-based=false, Temperature=02025.11 | 2.13 | 3.01 | |
| RESTModel=Llama-3.1-Instruct-70B, Training-based=false, Temperature=02025.11 | 2.03 | 2.34 | |
| RESTModel=Llama-3.1-Instruct-405B, Training-based=false, Temperature=12025.11 | 2.02 | 2.11 | |
| RESTModel=Llama-3.1-Instruct-70B, Training-based=false, Temperature=12025.11 | 1.94 | 2.24 | |
| SpSModel=Llama-3.1-Instruct-70B, Training-based=false, Temperature=02025.11 | 1.72 | 10.47 | |
| TokenRecyclingModel=Llama-3.1-Instruct-405B, Training-based=false, Temperature=12025.11 | 1.67 | 2.92 | |
| TokenRecyclingModel=Llama-3.1-Instruct-405B, Training-based=false, Temperature=02025.11 | 1.66 | 2.92 | |
| SpSModel=Llama-3.1-Instruct-70B, Training-based=false, Temperature=12025.11 | 1 | 6.08 |