Inference Efficiency on HumanEval
5.33Speedup FactorCATS w/ EAGLE
Evaluation Results
| Method | Links | ||||
|---|---|---|---|---|---|
| CATS w/ EAGLEModel=Llama2-13b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=tree base2026.05 | 5.33 | — | — | 5.3969 | |
| CATS w/ EAGLEModel=Llama2-7b, Decoding Strategy=Greedy, Decoding Structure=tree base2026.05 | 5.17 | — | — | 5.1741 | |
| FLyModel=Llama-3.1-Instruct-405B, Training-based=false, Temperature=02025.11 | 5.15 | — | — | 15.83 | |
| EAGLE-self, treeModel=Llama2-13b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=tree base2026.05 | 5.12 | — | — | 5.1758 | |
| CATSModel=Llama2-13b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=chain base2026.05 | 5.03 | — | — | 5.0678 | |
| FLyModel=Llama-3.1-Instruct-405B, Training-based=false, Temperature=12025.11 | 5 | — | — | 16.29 | |
| CATS w/ EAGLEModel=Llama2-7b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=tree base2026.05 | 4.99 | — | — | 4.9882 | |
| CATS w/ EAGLEModel=Llama2-13b, Decoding Strategy=Greedy, Decoding Structure=tree base2026.05 | 4.95 | — | — | 5.0178 | |
| CATSModel=Llama2-7b, Decoding Strategy=Greedy, Decoding Structure=chain base2026.05 | 4.72 | — | — | 4.724 | |
| EAGLE-selfModel=Llama2-13b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=chain base2026.05 | 4.69 | — | — | 4.7135 | |
| KangarooModel=Llama2-13b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=chain base2026.05 | 4.68 | — | — | 4.695 | |
| CATS w/ EAGLEModel=Vicuna-13b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=tree base2026.05 | 4.67 | — | — | 4.7269 | |
| CATSModel=Llama2-13b, Decoding Strategy=Greedy, Decoding Structure=chain base2026.05 | 4.66 | — | — | 4.6959 | |
| EAGLE-self, treeModel=Llama2-13b, Decoding Strategy=Greedy, Decoding Structure=tree base2026.05 | 4.65 | — | — | 4.6993 | |
| CATSModel=Llama2-7b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=chain base2026.05 | 4.44 | — | — | 4.4388 | |
| CATS w/ EAGLEModel=Vicuna-13b, Decoding Strategy=Greedy, Decoding Structure=tree base2026.05 | 4.19 | — | — | 4.2428 | |
| EAGLE-self, treeModel=Llama2-7b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=tree base2026.05 | 4.11 | — | — | 4.7703 | |
| KangarooModel=Llama2-13b, Decoding Strategy=Greedy, Decoding Structure=chain base2026.05 | 4.1 | — | — | 4.1133 | |
| EAGLE-selfModel=Llama2-13b, Decoding Strategy=Greedy, Decoding Structure=chain base2026.05 | 4.09 | — | — | 4.1082 | |
| EAGLE-3Model=Meta-Llama-3.3-Instruct-70B, Training-based=true, Temperature=02025.11 | 3.97 | — | — | 5.73 | |
| EAGLE-self, treeModel=Llama2-7b, Decoding Strategy=Greedy, Decoding Structure=tree base2026.05 | 3.95 | — | — | 4.5866 | |
| EAGLE-self, treeModel=Vicuna-13b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=tree base2026.05 | 3.94 | — | — | 3.9784 | |
| EAGLE-3Model=Meta-Llama-3.3-Instruct-70B, Training-based=true, Temperature=12025.11 | 3.74 | — | — | 5.4 | |
| KangarooModel=Llama2-7b, Decoding Strategy=Greedy, Decoding Structure=chain base2026.05 | 3.7 | — | — | 4.2946 | |
| EAGLE-selfModel=Llama2-7b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=chain base2026.05 | 3.68 | — | — | 4.2687 | |
| EAGLE-self, treeModel=Vicuna-13b, Decoding Strategy=Greedy, Decoding Structure=tree base2026.05 | 3.58 | — | — | 3.6168 | |
| EAGLE-selfModel=Llama2-7b, Decoding Strategy=Greedy, Decoding Structure=chain base2026.05 | 3.5 | — | — | 4.0588 | |
| KangarooModel=Llama2-7b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=chain base2026.05 | 3.5 | — | — | 4.0587 | |
| CATSModel=Vicuna-13b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=chain base2026.05 | 3.45 | — | — | 3.4792 | |
| SAM-Decoding[E2]Backbone=Vicuna-7B-v1.3, Auxiliary Method=EAGLE-2, Data Type=float16, Batch size=12024.11 | 3.35 | 4.95 | 130.28 | — | |
| KangarooModel=Vicuna-13b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=chain base2026.05 | 3.27 | — | — | 3.2753 | |
| DARTModel=Qwen4B, Temperature=02026.01 | 3.25 | — | — | — | |
| EAGLE-2Backbone=Vicuna-7B-v1.3, Data Type=float16, Batch size=12024.11 | 3.24 | 5.13 | 125.77 | — | |
| CATSModel=Vicuna-13b, Decoding Strategy=Greedy, Decoding Structure=chain base2026.05 | 3.2 | — | — | 3.2218 | |
| EAGLE-selfModel=Vicuna-13b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=chain base2026.05 | 3.2 | — | — | 3.2212 | |
| DARTModel=L2 7B, Temperature=02026.01 | 2.98 | — | — | — | |
| KangarooModel=Vicuna-13b, Decoding Strategy=Greedy, Decoding Structure=chain base2026.05 | 2.91 | — | — | 2.9183 | |
| EAGLE-selfModel=Vicuna-13b, Decoding Strategy=Greedy, Decoding Structure=chain base2026.05 | 2.9 | — | — | 2.9163 | |
| FLyModel=Llama-3.1-Instruct-70B, Training-based=false, Temperature=02025.11 | 2.86 | — | — | 12.61 | |
| SpSModel=Llama-3.1-Instruct-405B, Training-based=false, Temperature=02025.11 | 2.84 | — | — | 11.62 | |
| DARTModel=Qwen8B, Temperature=02026.01 | 2.8 | — | — | — | |
| DARTModel=Qwen1.7B, Temperature=02026.01 | 2.79 | — | — | — | |
| DARTModel=Qwen14B, Temperature=02026.01 | 2.79 | — | — | — | |
| FLyModel=Meta-Llama-3.3-Instruct-70B, Training-based=false, Temperature=02025.11 | 2.75 | — | — | 12.2 | |
| FLyModel=Meta-Llama-3.3-Instruct-70B, Training-based=false, Temperature=12025.11 | 2.75 | — | — | 12.55 | |
| HydraModel=L2 7B, Temperature=02026.01 | 2.6 | — | — | — | |
| FLyModel=Meta-Llama-3-Instruct-70B, Training-based=false, Temperature=02025.11 | 2.6 | — | — | 12.74 | |
| EAGLE-2Model=Meta-Llama-3-Instruct-70B, Training-based=true, Temperature=02025.11 | 2.56 | — | — | 4.24 | |
| FLyModel=Meta-Llama-3-Instruct-70B, Training-based=false, Temperature=12025.11 | 2.53 | — | — | 12.67 | |
| FLyModel=Llama-3.1-Instruct-70B, Training-based=false, Temperature=12025.11 | 2.52 | — | — | 11.96 | |
| MedusaModel=Vicuna-13b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=chain base2026.05 | 2.51 | — | — | 2.9535 | |
| DARTModel=Qwen14B, Temperature=12026.01 | 2.48 | — | — | — | |
| SAM-Decoding[T]Backbone=Vicuna-7B-v1.3, Auxiliary Method=Token Recycling, Data Type=float16, Batch size=12024.11 | 2.45 | 2.94 | 95.08 | — | |
| SpSModel=Llama-3.1-Instruct-405B, Training-based=false, Temperature=12025.11 | 2.39 | — | — | 9.78 | |
| MedusaModel=Vicuna-13b, Decoding Strategy=Greedy, Decoding Structure=chain base2026.05 | 2.39 | — | — | 2.8035 | |
| EAGLE-2Model=Meta-Llama-3-Instruct-70B, Training-based=true, Temperature=12025.11 | 2.38 | — | — | 3.95 | |
| EAGLE3Model=Qwen8B, Temperature=02026.01 | 2.37 | — | — | — | |
| DARTModel=Qwen32B, Temperature=02026.01 | 2.37 | — | — | — | |
| SAM-DecodingBackbone=Vicuna-7B-v1.3, Data Type=float16, Batch size=12024.11 | 2.29 | 2.64 | 88.91 | — | |
| MedusaModel=Llama2-7b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=chain base2026.05 | 2.28 | — | — | 2.2848 | |
| EAGLE3Model=Qwen14B, Temperature=02026.01 | 2.27 | — | — | — | |
| MedusaModel=L2 7B, Temperature=02026.01 | 2.22 | — | — | — | |
| EAGLE3Model=Qwen4B, Temperature=02026.01 | 2.2 | — | — | — | |
| EAGLE3Model=Qwen32B, Temperature=02026.01 | 2.19 | — | — | — | |
| DARTModel=Qwen32B, Temperature=12026.01 | 2.19 | — | — | — | |
| EAGLE3Model=Qwen14B, Temperature=12026.01 | 2.17 | — | — | — | |
| TokenRecyclingModel=Llama-3.1-Instruct-70B, Training-based=false, Temperature=12025.11 | 2.15 | — | — | 3.06 | |
| RESTModel=Llama-3.1-Instruct-405B, Training-based=false, Temperature=02025.11 | 2.14 | — | — | 2.41 | |
| TokenRecyclingModel=Llama-3.1-Instruct-70B, Training-based=false, Temperature=02025.11 | 2.12 | — | — | 3.07 | |
| EAGLE3Model=Qwen32B, Temperature=12026.01 | 2.08 | — | — | — | |
| MedusaModel=Llama2-7b, Decoding Strategy=Greedy, Decoding Structure=chain base2026.05 | 1.97 | — | — | 1.9657 | |
| MedusaModel=Llama2-13b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=chain base2026.05 | 1.95 | — | — | 2.2848 | |
| Token RecyclingBackbone=Vicuna-7B-v1.3, Data Type=float16, Batch size=12024.11 | 1.94 | 2.78 | 75.44 | — | |
| EAGLE3Model=Qwen1.7B, Temperature=02026.01 | 1.94 | — | — | — | |
| RESTModel=Llama-3.1-Instruct-70B, Training-based=false, Temperature=02025.11 | 1.89 | — | — | 2.22 | |
| PLDModel=L2 7B, Temperature=02026.01 | 1.84 | — | — | — | |
| RESTModel=Llama-3.1-Instruct-405B, Training-based=false, Temperature=12025.11 | 1.83 | — | — | 2.07 | |
| RESTModel=Llama-3.1-Instruct-70B, Training-based=false, Temperature=12025.11 | 1.81 | — | — | 2.13 | |
| REST*Backbone=Vicuna-7B-v1.3, Data Type=float16, Batch size=12024.11 | 1.74 | 1.85 | 34.6 | — | |
| PIABackbone=Vicuna-7B-v1.3, Data Type=float16, Batch size=12024.11 | 1.68 | 2.62 | 65.49 | — | |
| MedusaModel=Llama2-13b, Decoding Strategy=Greedy, Decoding Structure=chain base2026.05 | 1.65 | — | — | 1.9425 | |
| LookaheadModel=L2 7B, Temperature=02026.01 | 1.64 | — | — | — | |
| SpSModel=Llama-3.1-Instruct-70B, Training-based=false, Temperature=02025.11 | 1.64 | — | — | 9.89 | |
| TokenRecyclingModel=Llama-3.1-Instruct-405B, Training-based=false, Temperature=12025.11 | 1.56 | — | — | 2.91 | |
| Lookahead*Backbone=Vicuna-7B-v1.3, Data Type=float16, Batch size=12024.11 | 1.54 | 1.76 | 30.81 | — | |
| TokenRecyclingModel=Llama-3.1-Instruct-405B, Training-based=false, Temperature=02025.11 | 1.54 | — | — | 2.88 | |
| PLDBackbone=Vicuna-7B-v1.3, Data Type=float16, Batch size=12024.11 | 1.52 | 1.65 | 59.04 | — | |
| SpSModel=Llama-3.1-Instruct-70B, Training-based=false, Temperature=12025.11 | 1.07 | — | — | 6.29 | |
| SPSModel=Qwen32B, Temperature=02026.01 | 1.06 | — | — | — | |
| SPSModel=Qwen14B, Temperature=02026.01 | 0.96 | — | — | — |