Inference Acceleration on Spec-Bench
5.31SpeedupCATS w/ EAGLE
Evaluation results
| Method | Links | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| CATS w/ EAGLEModel=Llama2-7b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=tree base2026.05 | 5.31 | — | — | — | — | — | — | — | — | — | — | 5.3143 | |
| CATS w/ EAGLEModel=Llama2-13b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=tree base2026.05 | 5.01 | — | — | — | — | — | — | — | — | — | — | 5.0813 | |
| CATSModel=Llama2-7b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=chain base2026.05 | 4.88 | — | — | — | — | — | — | — | — | — | — | 4.8831 | |
| CATS w/ EAGLEModel=Llama2-7b, Decoding Strategy=Greedy, Decoding Structure=tree base2026.05 | 4.83 | — | — | — | — | — | — | — | — | — | — | 4.8318 | |
| EAGLE-self, treeModel=Llama2-13b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=tree base2026.05 | 4.67 | — | — | — | — | — | — | — | — | — | — | 4.7176 | |
| CATSModel=Llama2-13b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=chain base2026.05 | 4.61 | — | — | — | — | — | — | — | — | — | — | 4.6461 | |
| CATS w/ EAGLEModel=Llama2-13b, Decoding Strategy=Greedy, Decoding Structure=tree base2026.05 | 4.51 | — | — | — | — | — | — | — | — | — | — | 4.5707 | |
| CATSModel=Llama2-7b, Decoding Strategy=Greedy, Decoding Structure=chain base2026.05 | 4.33 | — | — | — | — | — | — | — | — | — | — | 4.3335 | |
| EAGLE-self, treeModel=Llama2-7b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=tree base2026.05 | 4.29 | — | — | — | — | — | — | — | — | — | — | 4.9713 | |
| KangarooModel=Llama2-13b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=chain base2026.05 | 4.14 | — | — | — | — | — | — | — | — | — | — | 4.1486 | |
| EAGLE-selfModel=Llama2-13b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=chain base2026.05 | 4.12 | — | — | — | — | — | — | — | — | — | — | 4.1452 | |
| EAGLE-self, treeModel=Llama2-13b, Decoding Strategy=Greedy, Decoding Structure=tree base2026.05 | 4.04 | — | — | — | — | — | — | — | — | — | — | 4.0829 | |
| CATSModel=Llama2-13b, Decoding Strategy=Greedy, Decoding Structure=chain base2026.05 | 4.02 | — | — | — | — | — | — | — | — | — | — | 4.057 | |
| CATS w/ EAGLEModel=Vicuna-13b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=tree base2026.05 | 3.93 | — | — | — | — | — | — | — | — | — | — | 3.9851 | |
| KangarooModel=Llama2-7b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=chain base2026.05 | 3.87 | — | — | — | — | — | — | — | — | — | — | 4.4849 | |
| EAGLE-selfModel=Llama2-7b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=chain base2026.05 | 3.87 | — | — | — | — | — | — | — | — | — | — | 4.491 | |
| EAGLE-self, treeModel=Llama2-7b, Decoding Strategy=Greedy, Decoding Structure=tree base2026.05 | 3.75 | — | — | — | — | — | — | — | — | — | — | 4.3529 | |
| KangarooModel=Llama2-13b, Decoding Strategy=Greedy, Decoding Structure=chain base2026.05 | 3.53 | — | — | — | — | — | — | — | — | — | — | 3.5409 | |
| EAGLE-selfModel=Llama2-13b, Decoding Strategy=Greedy, Decoding Structure=chain base2026.05 | 3.53 | — | — | — | — | — | — | — | — | — | — | 3.5478 | |
| CATS w/ EAGLEModel=Vicuna-13b, Decoding Strategy=Greedy, Decoding Structure=tree base2026.05 | 3.42 | — | — | — | — | — | — | — | — | — | — | 3.4708 | |
| KangarooModel=Llama2-7b, Decoding Strategy=Greedy, Decoding Structure=chain base2026.05 | 3.29 | — | — | — | — | — | — | — | — | — | — | 3.8176 | |
| EAGLE-selfModel=Llama2-7b, Decoding Strategy=Greedy, Decoding Structure=chain base2026.05 | 3.28 | — | — | — | — | — | — | — | — | — | — | 3.8068 | |
| EAGLE-self, treeModel=Vicuna-13b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=tree base2026.05 | 3.16 | — | — | — | — | — | — | — | — | — | — | 3.1934 | |
| DECODevice=Jetson AGX 64GB, Kernel=Ours (acceleration kernel)2026.05 | 3 | 44.15 | 44.08 | 42.78 | 45.29 | 45.99 | 43.64 | — | — | — | 44.32 | — | |
| CATSModel=Vicuna-13b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=chain base2026.05 | 2.95 | — | — | — | — | — | — | — | — | — | — | 2.9743 | |
| EAGLE-self, treeModel=Vicuna-13b, Decoding Strategy=Greedy, Decoding Structure=tree base2026.05 | 2.77 | — | — | — | — | — | — | — | — | — | — | 2.8035 | |
| CATSModel=Vicuna-13b, Decoding Strategy=Greedy, Decoding Structure=chain base2026.05 | 2.64 | — | — | — | — | — | — | — | — | — | — | 2.6588 | |
| SAM-Decoding[E2]Backbone=Vicuna-7B-v1.3, Auxiliary Method=EAGLE-2, Data Type=float16, Batch size=12024.11 | 2.58 | — | — | — | — | — | — | 4.62 | 97.56 | — | — | — | |
| DECODevice=RTX 4090 24GB, Kernel=Ours (acceleration kernel)2026.05 | 2.58 | 224.28 | 222.65 | 218.24 | 228.57 | 231.88 | 222.19 | — | — | — | 224.63 | — | |
| EAGLE-selfModel=Vicuna-13b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=chain base2026.05 | 2.5 | — | — | — | — | — | — | — | — | — | — | 2.5144 | |
| KangarooModel=Vicuna-13b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=chain base2026.05 | 2.48 | — | — | — | — | — | — | — | — | — | — | 2.4889 | |
| EAGLE-2Backbone=Vicuna-7B-v1.3, Data Type=float16, Batch size=12024.11 | 2.38 | — | — | — | — | — | — | 4.36 | 90.14 | — | — | — | |
| SAM-Decoding[T]Backbone=Vicuna-7B-v1.3, Auxiliary Method=Token Recycling, Data Type=float16, Batch size=12024.11 | 2.27 | — | — | — | — | — | — | 3.03 | 85.73 | — | — | — | |
| MedusaModel=Vicuna-13b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=chain base2026.05 | 2.2 | — | — | — | — | — | — | — | — | — | — | 2.5852 | |
| KangarooModel=Vicuna-13b, Decoding Strategy=Greedy, Decoding Structure=chain base2026.05 | 2.19 | — | — | — | — | — | — | — | — | — | — | 2.1937 | |
| EAGLE-selfModel=Vicuna-13b, Decoding Strategy=Greedy, Decoding Structure=chain base2026.05 | 2.18 | — | — | — | — | — | — | — | — | — | — | 2.1952 | |
| SAM-Decoding[E2]Backbone=Llama3-8B, Combined Method=EAGLE-22024.11 | 2.08 | 2.36 | 1.96 | 1.98 | 1.79 | 2.32 | 2.11 | 3.92 | 72.47 | — | — | — | |
| SAM-Decoding[T]Backbone=Llama3-8B, Combined Method=Token Recycling2024.11 | 2.05 | 2.09 | 1.93 | 2.04 | 1.82 | 2.32 | 2.12 | 2.63 | 71.73 | — | — | — | |
| MedusaModel=Vicuna-13b, Decoding Strategy=Greedy, Decoding Structure=chain base2026.05 | 2.03 | — | — | — | — | — | — | — | — | — | — | 2.392 | |
| EAGLE-2Backbone=Llama3-8B2024.11 | 1.98 | 2.08 | 1.95 | 1.85 | 1.8 | 2.31 | 1.87 | 3.9 | 68.69 | — | — | — | |
| Token RecyclingBackbone=Llama3-8B2024.11 | 1.91 | 1.92 | 1.88 | 1.85 | 1.75 | 2.24 | 1.82 | 2.76 | 66.42 | — | — | — | |
| SAM-DecodingBackbone=Vicuna-7B-v1.3, Data Type=float16, Batch size=12024.11 | 1.84 | — | — | — | — | — | — | 2.3 | 69.37 | — | — | — | |
| Token RecyclingBackbone=Vicuna-7B-v1.3, Data Type=float16, Batch size=12024.11 | 1.84 | — | — | — | — | — | — | 2.83 | 69.65 | — | — | — | |
| MedusaModel=Llama2-7b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=chain base2026.05 | 1.75 | — | — | — | — | — | — | — | — | — | — | 1.7482 | |
| MedusaModel=Llama2-7b, Decoding Strategy=Greedy, Decoding Structure=chain base2026.05 | 1.67 | — | — | — | — | — | — | — | — | — | — | 1.666 | |
| MedusaModel=Llama2-13b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=chain base2026.05 | 1.64 | — | — | — | — | — | — | — | — | — | — | 1.9322 | |
| PLDBackbone=Vicuna-7B-v1.3, Data Type=float16, Batch size=12024.11 | 1.56 | — | — | — | — | — | — | 1.75 | 59.02 | — | — | — | |
| MedusaModel=Llama2-13b, Decoding Strategy=Greedy, Decoding Structure=chain base2026.05 | 1.55 | — | — | — | — | — | — | — | — | — | — | 1.825 | |
| SAM-DecodingBackbone=Llama3-8B2024.11 | 1.51 | 1.59 | 1.35 | 1.5 | 1.35 | 1.54 | 1.75 | 1.72 | 52.35 | — | — | — | |
| PIABackbone=Vicuna-7B-v1.3, Data Type=float16, Batch size=12024.11 | 1.47 | — | — | — | — | — | — | 2.08 | 55.45 | — | — | — | |
| REST*Backbone=Vicuna-7B-v1.3, Data Type=float16, Batch size=12024.11 | 1.38 | — | — | — | — | — | — | 1.63 | 51.34 | — | — | — | |
| PLDBackbone=Llama3-8B2024.11 | 1.28 | 1.3 | 1.12 | 1.41 | 1.03 | 1.3 | 1.53 | 1.39 | 44.26 | — | — | — | |
| Lookahead*Backbone=Vicuna-7B-v1.3, Data Type=float16, Batch size=12024.11 | 1.2 | — | — | — | — | — | — | 1.63 | 44.37 | — | — | — | |
| ARParameters=7B2024.08 | — | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 54.3 | — | — | — | |
| ARParameters=13B2024.08 | — | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 39.41 | — | — | — | |
| ARParameters=33B2024.08 | — | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 18.44 | — | — | — | |
| Baseline ARDevice=RTX 4090 24GB, Kernel=Standard autoregressive decoding2026.05 | — | 86.63 | 87.81 | 86.2 | 87.88 | 87.83 | 86.24 | — | — | — | 87.1 | — | |
| Baseline ARDevice=Jetson AGX 64GB, Kernel=Standard autoregressive decoding2026.05 | — | 14.68 | 14.89 | 14.61 | 14.91 | 14.89 | 14.62 | — | — | — | 14.77 | — | |
| EAGLE-2Model=Vicuna-33B2024.11 | — | 3.29 | 2.31 | 2.73 | 2.51 | 3.65 | 2.46 | 4.06 | 25.86 | 2.83 | — | — | |
| LadeParameters=7B2024.08 | — | 1.42 | 1.12 | 1.21 | 1.21 | 1.52 | 1.13 | 1.64 | 69.03 | — | — | — | |
| LadeParameters=13B2024.08 | — | 1.29 | 1.06 | 1.16 | 1.12 | 1.48 | 1.09 | 1.63 | 47.5 | — | — | — | |
| LadeParameters=33B2024.08 | — | 1.32 | 1.09 | 1.2 | 1.17 | 1.55 | 1.14 | 1.61 | 23.03 | — | — | — | |
| MedusaParameters=7B2024.08 | — | 1.9 | 1.57 | 1.48 | 1.58 | 1.87 | 1.45 | 2.31 | 89.41 | — | — | — | |
| MedusaParameters=13B2024.08 | — | 1.94 | 1.66 | 1.57 | 1.62 | 1.98 | 1.53 | 2.39 | 67.92 | — | — | — | |
| MedusaParameters=33B2024.08 | — | 1.98 | 1.75 | 1.63 | 1.68 | 2.09 | 1.61 | 2.32 | 33.11 | — | — | — | |
| PLDModel=Vicuna-33B2024.11 | — | 1.5 | 1.07 | 2.06 | 1.09 | 1.59 | 1.51 | 1.65 | 13.33 | 1.46 | — | — | |
| PLDParameters=7B2024.08 | — | 1.53 | 0.98 | 2.36 | 1.1 | 1.5 | 1.74 | 1.75 | 83.3 | — | — | — | |
| PLDParameters=13B2024.08 | — | 1.45 | 1.01 | 2.1 | 1.02 | 1.55 | 1.65 | 1.67 | 57.01 | — | — | — | |
| PLDParameters=33B2024.08 | — | 1.43 | 1.06 | 1.94 | 1.08 | 1.55 | 1.41 | 1.55 | 25.89 | — | — | — | |
| RESTParameters=7B2024.08 | — | 1.37 | 1.05 | 1.12 | 1.42 | 1.06 | 1.3 | 1.84 | 66.29 | — | — | — | |
| RESTParameters=13B2024.08 | — | 1.51 | 1.14 | 1.31 | 1.5 | 1.17 | 1.5 | 1.82 | 53.34 | — | — | — | |
| RESTParameters=33B2024.08 | — | 1.63 | 1.27 | 1.42 | 1.61 | 1.29 | 1.57 | 1.81 | 26.99 | — | — | — | |
| SAM-DecodingModel=Vicuna-33B2024.11 | — | 1.91 | 1.25 | 1.98 | 1.48 | 1.83 | 1.66 | 1.97 | 15.35 | 1.68 | — | — | |
| SAM-DecodingModel=Vicuna-33B, variant=T2024.11 | — | 2.31 | 1.79 | 2.53 | 1.9 | 2.48 | 2.06 | 2.68 | 19.87 | 2.18 | — | — | |
| SAM-DecodingModel=Vicuna-33B, variant=E22024.11 | — | 3.4 | 2.25 | 2.93 | 2.43 | 3.45 | 2.54 | 4.08 | 25.91 | 2.84 | — | — | |
| Token RecyclingModel=Vicuna-33B2024.11 | — | 2.1 | 1.84 | 2.19 | 1.88 | 2.42 | 1.92 | 2.7 | 18.8 | 2.06 | — | — | |
| TRParameters=7B2024.08 | — | 2.17 | 1.9 | 1.94 | 1.95 | 2.4 | 1.78 | 2.7 | 110.06 | — | — | — | |
| TRParameters=13B2024.08 | — | 1.98 | 1.77 | 1.89 | 1.75 | 2.21 | 1.73 | 2.72 | 74.57 | — | — | — | |
| TRParameters=33B2024.08 | — | 1.95 | 1.75 | 1.92 | 1.77 | 2.24 | 1.78 | 2.63 | 35.16 | — | — | — |