LLM Inference Acceleration on GSM8K
5.51SpeedupCATS w/ EAGLE
Evaluation Results
| Method | Links | ||||
|---|---|---|---|---|---|
| CATS w/ EAGLEModel=Llama2-7b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=tree base2026.05 | 5.51 | 5.5096 | — | — | |
| CATS w/ EAGLEModel=Llama2-13b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=tree base2026.05 | 5.48 | 5.5524 | — | — | |
| CATS w/ EAGLEModel=Llama2-7b, Decoding Strategy=Greedy, Decoding Structure=tree base2026.05 | 5.42 | 5.4185 | — | — | |
| CATSModel=Llama2-13b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=chain base2026.05 | 5.27 | 5.3102 | — | — | |
| CATSModel=Llama2-7b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=chain base2026.05 | 5.19 | 5.187 | — | — | |
| CATSModel=Llama2-7b, Decoding Strategy=Greedy, Decoding Structure=chain base2026.05 | 5.1 | 5.1055 | — | — | |
| EAGLE-self, treeModel=Llama2-13b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=tree base2026.05 | 5.1 | 5.1449 | — | — | |
| FLyModel=Llama-3.1-Instruct-405B, Training-based=false, Temperature=12025.11 | 4.97 | 19.4 | — | — | |
| CATS w/ EAGLEModel=Llama2-13b, Decoding Strategy=Greedy, Decoding Structure=tree base2026.05 | 4.79 | 4.8534 | — | — | |
| FLyModel=Llama-3.1-Instruct-405B, Training-based=false, Temperature=02025.11 | 4.61 | 17.02 | — | — | |
| KangarooModel=Llama2-13b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=chain base2026.05 | 4.59 | 4.6031 | — | — | |
| EAGLE-selfModel=Llama2-13b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=chain base2026.05 | 4.57 | 4.5883 | — | — | |
| EAGLE-self, treeModel=Llama2-7b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=tree base2026.05 | 4.55 | 5.2814 | — | — | |
| EAGLE-self, treeModel=Llama2-7b, Decoding Strategy=Greedy, Decoding Structure=tree base2026.05 | 4.39 | 5.0954 | — | — | |
| CATSModel=Llama2-13b, Decoding Strategy=Greedy, Decoding Structure=chain base2026.05 | 4.31 | 4.3415 | — | — | |
| EAGLE-self, treeModel=Llama2-13b, Decoding Strategy=Greedy, Decoding Structure=tree base2026.05 | 4.24 | 4.2793 | — | — | |
| EAGLE-selfModel=Llama2-7b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=chain base2026.05 | 4.24 | 4.9139 | — | — | |
| KangarooModel=Llama2-13b, Decoding Strategy=Greedy, Decoding Structure=chain base2026.05 | 4.23 | 4.2421 | — | — | |
| KangarooModel=Llama2-7b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=chain base2026.05 | 4.18 | 4.8506 | — | — | |
| EAGLE-selfModel=Llama2-7b, Decoding Strategy=Greedy, Decoding Structure=chain base2026.05 | 4.1 | 4.7597 | — | — | |
| CATS w/ EAGLEModel=Vicuna-13b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=tree base2026.05 | 4.09 | 4.1399 | — | — | |
| EAGLE-selfModel=Llama2-13b, Decoding Strategy=Greedy, Decoding Structure=chain base2026.05 | 3.84 | 3.8572 | — | — | |
| KangarooModel=Llama2-7b, Decoding Strategy=Greedy, Decoding Structure=chain base2026.05 | 3.82 | 4.4296 | — | — | |
| EAGLE-3Model=Meta-Llama-3.3-Instruct-70B, Training-based=true, Temperature=02025.11 | 3.72 | 5.64 | — | — | |
| EAGLE-3Model=Meta-Llama-3.3-Instruct-70B, Training-based=true, Temperature=12025.11 | 3.5 | 5.31 | — | — | |
| CATS w/ EAGLEModel=Vicuna-13b, Decoding Strategy=Greedy, Decoding Structure=tree base2026.05 | 3.5 | 3.5481 | — | — | |
| EAGLE-self, treeModel=Vicuna-13b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=tree base2026.05 | 3.33 | 3.3631 | — | — | |
| CATSModel=Vicuna-13b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=chain base2026.05 | 3.05 | 3.0736 | — | — | |
| FLyModel=Llama-3.1-Instruct-70B, Training-based=false, Temperature=02025.11 | 2.98 | 12.57 | — | — | |
| FLyModel=Meta-Llama-3-Instruct-70B, Training-based=false, Temperature=02025.11 | 2.94 | 11.67 | — | — | |
| FLyModel=Meta-Llama-3-Instruct-70B, Training-based=false, Temperature=12025.11 | 2.91 | 11.55 | — | — | |
| FLyModel=Meta-Llama-3.3-Instruct-70B, Training-based=false, Temperature=12025.11 | 2.85 | 12.4 | — | — | |
| EAGLE-self, treeModel=Vicuna-13b, Decoding Strategy=Greedy, Decoding Structure=tree base2026.05 | 2.85 | 2.876 | — | — | |
| KangarooModel=Vicuna-13b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=chain base2026.05 | 2.76 | 2.7695 | — | — | |
| CATSModel=Vicuna-13b, Decoding Strategy=Greedy, Decoding Structure=chain base2026.05 | 2.74 | 2.7636 | — | — | |
| FLyModel=Meta-Llama-3.3-Instruct-70B, Training-based=false, Temperature=02025.11 | 2.68 | 11.65 | — | — | |
| FLyModel=Llama-3.1-Instruct-70B, Training-based=false, Temperature=12025.11 | 2.67 | 11.93 | — | — | |
| SpSModel=Llama-3.1-Instruct-405B, Training-based=false, Temperature=02025.11 | 2.59 | 10.98 | — | — | |
| EAGLE-selfModel=Vicuna-13b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=chain base2026.05 | 2.58 | 2.5893 | — | — | |
| EAGLE-2Model=Meta-Llama-3-Instruct-70B, Training-based=true, Temperature=02025.11 | 2.54 | 3.74 | — | — | |
| KangarooModel=Vicuna-13b, Decoding Strategy=Greedy, Decoding Structure=chain base2026.05 | 2.38 | 2.3848 | — | — | |
| MedusaModel=Vicuna-13b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=chain base2026.05 | 2.38 | 2.792 | — | — | |
| EAGLE-2Model=Meta-Llama-3-Instruct-70B, Training-based=true, Temperature=12025.11 | 2.36 | 3.48 | — | — | |
| MedusaModel=Vicuna-13b, Decoding Strategy=Greedy, Decoding Structure=chain base2026.05 | 2.2 | 2.5824 | — | — | |
| EAGLE-selfModel=Vicuna-13b, Decoding Strategy=Greedy, Decoding Structure=chain base2026.05 | 2.19 | 2.2031 | — | — | |
| MedusaModel=Llama2-7b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=chain base2026.05 | 2.19 | 2.1875 | — | — | |
| SpSModel=Llama-3.1-Instruct-405B, Training-based=false, Temperature=12025.11 | 2.18 | 9.25 | — | — | |
| SpSModel=Llama-3.1-Instruct-70B, Training-based=false, Temperature=02025.11 | 2.13 | 9.87 | — | — | |
| SpSModel=Llama-3.1-Instruct-70B, Training-based=false, Temperature=12025.11 | 2.13 | 9.7 | — | — | |
| TokenRecyclingModel=Llama-3.1-Instruct-70B, Training-based=false, Temperature=02025.11 | 2.1 | 3.04 | — | — | |
| TokenRecyclingModel=Llama-3.1-Instruct-70B, Training-based=false, Temperature=12025.11 | 2.08 | 3.02 | — | — | |
| RESTModel=Llama-3.1-Instruct-405B, Training-based=false, Temperature=02025.11 | 2.07 | 2.13 | — | — | |
| MedusaModel=Llama2-13b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=chain base2026.05 | 1.92 | 2.2499 | — | — | |
| RESTModel=Llama-3.1-Instruct-70B, Training-based=false, Temperature=02025.11 | 1.79 | 2 | — | — | |
| RESTModel=Llama-3.1-Instruct-405B, Training-based=false, Temperature=12025.11 | 1.78 | 1.82 | — | — | |
| ReMoEHardware=Jetson Orin NX 16GB, Backend=llama.cpp, Number of parallel requests (-np)=1, Sequence length (-n)=128, Memory mapping (--mmap)=true2026.05 | 1.77 | — | 4,736.7 | 346.04 | |
| MedusaModel=Llama2-7b, Decoding Strategy=Greedy, Decoding Structure=chain base2026.05 | 1.74 | 1.7358 | — | — | |
| RESTModel=Llama-3.1-Instruct-70B, Training-based=false, Temperature=12025.11 | 1.71 | 1.91 | — | — | |
| MedusaModel=Llama2-13b, Decoding Strategy=Greedy, Decoding Structure=chain base2026.05 | 1.68 | 1.9721 | — | — | |
| TokenRecyclingModel=Llama-3.1-Instruct-405B, Training-based=false, Temperature=12025.11 | 1.48 | 2.85 | — | — | |
| TokenRecyclingModel=Llama-3.1-Instruct-405B, Training-based=false, Temperature=02025.11 | 1.47 | 2.83 | — | — | |
| BaselineHardware=Jetson Orin NX 16GB, Backend=llama.cpp, Number of parallel requests (-np)=1, Sequence length (-n)=128, Memory mapping (--mmap)=true2026.05 | — | — | 6,041.65 | 613.73 |