LLM Inference on Alpaca
5.56SpeedupCATS w/ EAGLE
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| CATS w/ EAGLEModel=Llama2-7b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=tree base2026.05 | 5.56 | 5.5601 | — | — | — | — | |
| CATS w/ EAGLEModel=Llama2-13b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=tree base2026.05 | 5.27 | 5.3241 | — | — | — | — | |
| CATSModel=Llama2-7b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=chain base2026.05 | 5.2 | 5.1976 | — | — | — | — | |
| EAGLE-self, treeModel=Llama2-13b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=tree base2026.05 | 4.99 | 5.0338 | — | — | — | — | |
| CATSModel=Llama2-13b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=chain base2026.05 | 4.92 | 4.9464 | — | — | — | — | |
| CATS w/ EAGLEModel=Llama2-7b, Decoding Strategy=Greedy, Decoding Structure=tree base2026.05 | 4.91 | 4.9101 | — | — | — | — | |
| CATS w/ EAGLEModel=Llama2-13b, Decoding Strategy=Greedy, Decoding Structure=tree base2026.05 | 4.75 | 4.7998 | — | — | — | — | |
| KangarooModel=Llama2-13b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=chain base2026.05 | 4.73 | 4.7376 | — | — | — | — | |
| EAGLE-selfModel=Llama2-13b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=chain base2026.05 | 4.72 | 4.7401 | — | — | — | — | |
| EAGLE-self, treeModel=Llama2-7b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=tree base2026.05 | 4.59 | 5.3228 | — | — | — | — | |
| CATSModel=Llama2-7b, Decoding Strategy=Greedy, Decoding Structure=chain base2026.05 | 4.46 | 4.4642 | — | — | — | — | |
| EAGLE-self, treeModel=Llama2-13b, Decoding Strategy=Greedy, Decoding Structure=tree base2026.05 | 4.42 | 4.4566 | — | — | — | — | |
| CATSModel=Llama2-13b, Decoding Strategy=Greedy, Decoding Structure=chain base2026.05 | 4.34 | 4.3696 | — | — | — | — | |
| EAGLE-selfModel=Llama2-7b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=chain base2026.05 | 4.3 | 4.9867 | — | — | — | — | |
| KangarooModel=Llama2-7b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=chain base2026.05 | 4.29 | 4.9796 | — | — | — | — | |
| CATS w/ EAGLEModel=Vicuna-13b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=tree base2026.05 | 4.02 | 4.0683 | — | — | — | — | |
| KangarooModel=Llama2-13b, Decoding Strategy=Greedy, Decoding Structure=chain base2026.05 | 3.97 | 3.9754 | — | — | — | — | |
| EAGLE-selfModel=Llama2-13b, Decoding Strategy=Greedy, Decoding Structure=chain base2026.05 | 3.97 | 3.9805 | — | — | — | — | |
| EAGLE-self, treeModel=Llama2-7b, Decoding Strategy=Greedy, Decoding Structure=tree base2026.05 | 3.91 | 4.5393 | — | — | — | — | |
| KangarooModel=Llama2-7b, Decoding Strategy=Greedy, Decoding Structure=chain base2026.05 | 3.49 | 4.0431 | — | — | — | — | |
| EAGLE-selfModel=Llama2-7b, Decoding Strategy=Greedy, Decoding Structure=chain base2026.05 | 3.49 | 4.0429 | — | — | — | — | |
| CATS w/ EAGLEModel=Vicuna-13b, Decoding Strategy=Greedy, Decoding Structure=tree base2026.05 | 3.32 | 3.3568 | — | — | — | — | |
| EAGLE-self, treeModel=Vicuna-13b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=tree base2026.05 | 3.28 | 3.3051 | — | — | — | — | |
| CATSModel=Vicuna-13b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=chain base2026.05 | 2.98 | 2.9963 | — | — | — | — | |
| DARTModel=L2 7B, Temperature=02026.01 | 2.95 | — | — | — | — | — | |
| EAGLE-self, treeModel=Vicuna-13b, Decoding Strategy=Greedy, Decoding Structure=tree base2026.05 | 2.81 | 2.8337 | — | — | — | — | |
| DARTModel=Qwen14B, Temperature=02026.01 | 2.73 | — | — | — | — | — | |
| HydraModel=L2 7B, Temperature=02026.01 | 2.72 | — | — | — | — | — | |
| EAGLE-selfModel=Vicuna-13b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=chain base2026.05 | 2.65 | 2.6609 | — | — | — | — | |
| DARTModel=Qwen1.7B, Temperature=02026.01 | 2.6 | — | — | — | — | — | |
| KangarooModel=Vicuna-13b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=chain base2026.05 | 2.59 | 2.5927 | — | — | — | — | |
| DARTModel=Qwen4B, Temperature=02026.01 | 2.55 | — | — | — | — | — | |
| CATSModel=Vicuna-13b, Decoding Strategy=Greedy, Decoding Structure=chain base2026.05 | 2.53 | 2.5493 | — | — | — | — | |
| DARTModel=Qwen8B, Temperature=02026.01 | 2.51 | — | — | — | — | — | |
| DARTModel=Qwen14B, Temperature=12026.01 | 2.38 | — | — | — | — | — | |
| MedusaModel=Vicuna-13b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=chain base2026.05 | 2.31 | 2.7148 | — | — | — | — | |
| KangarooModel=Vicuna-13b, Decoding Strategy=Greedy, Decoding Structure=chain base2026.05 | 2.22 | 2.2226 | — | — | — | — | |
| EAGLE-selfModel=Vicuna-13b, Decoding Strategy=Greedy, Decoding Structure=chain base2026.05 | 2.21 | 2.2183 | — | — | — | — | |
| EAGLE3Model=Qwen4B, Temperature=02026.01 | 2.15 | — | — | — | — | — | |
| MedusaModel=L2 7B, Temperature=02026.01 | 2.13 | — | — | — | — | — | |
| MedusaModel=Vicuna-13b, Decoding Strategy=Greedy, Decoding Structure=chain base2026.05 | 2.04 | 2.3959 | — | — | — | — | |
| DARTModel=Qwen32B, Temperature=02026.01 | 2.03 | — | — | — | — | — | |
| EAGLE3Model=Qwen8B, Temperature=02026.01 | 2.02 | — | — | — | — | — | |
| MedusaModel=Llama2-7b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=chain base2026.05 | 1.9 | 1.8955 | — | — | — | — | |
| EAGLE3Model=Qwen1.7B, Temperature=02026.01 | 1.84 | — | — | — | — | — | |
| DARTModel=Qwen32B, Temperature=12026.01 | 1.84 | — | — | — | — | — | |
| MedusaModel=Llama2-7b, Decoding Strategy=Greedy, Decoding Structure=chain base2026.05 | 1.78 | 1.7758 | — | — | — | — | |
| MedusaModel=Llama2-13b, Decoding Strategy=Relaxed (temperature = 0.7), Decoding Structure=chain base2026.05 | 1.76 | 2.0718 | — | — | — | — | |
| EAGLE3Model=Qwen32B, Temperature=02026.01 | 1.72 | — | — | — | — | — | |
| EAGLE3Model=Qwen14B, Temperature=02026.01 | 1.69 | — | — | — | — | — | |
| MedusaModel=Llama2-13b, Decoding Strategy=Greedy, Decoding Structure=chain base2026.05 | 1.65 | 1.9425 | — | — | — | — | |
| EAGLE3Model=Qwen14B, Temperature=12026.01 | 1.62 | — | — | — | — | — | |
| EAGLE3Model=Qwen32B, Temperature=12026.01 | 1.57 | — | — | — | — | — | |
| LookaheadModel=L2 7B, Temperature=02026.01 | 1.42 | — | — | — | — | — | |
| PLDModel=L2 7B, Temperature=02026.01 | 1.18 | — | — | — | — | — | |
| SPSModel=Qwen32B, Temperature=02026.01 | 1.07 | — | — | — | — | — | |
| SPSModel=Qwen14B, Temperature=02026.01 | 1.05 | — | — | — | — | — | |
| BUDDYRM Blocks=4(12.5%)2026.06 | — | — | 1,991.54 | 1.14 | 40.03 | 1.01 | |
| BUDDYRM Blocks=8(25%)2026.06 | — | — | 2,440.14 | 1.39 | 46.8 | 1.19 | |
| BUDDYRM Blocks=12(37.5%)2026.06 | — | — | 2,548.39 | 1.45 | 51.49 | 1.3 | |
| BUDDYRM Blocks=16(50%)2026.06 | — | — | 3,348.18 | 1.91 | 64.84 | 1.64 | |
| DenseRM Blocks=0(0.0%)2026.06 | — | — | 1,753.16 | 1 | 39.48 | 1 | |
| FiRSTRM Blocks=4(12.5%)2026.06 | — | — | 1,794.51 | 1.02 | 33.46 | 0.85 | |
| FiRSTRM Blocks=8(25%)2026.06 | — | — | 1,795.22 | 1.02 | 33.42 | 0.85 | |
| FiRSTRM Blocks=12(37.5%)2026.06 | — | — | 1,780.78 | 1.02 | 33.3 | 0.84 | |
| FiRSTRM Blocks=16(50%)2026.06 | — | — | 1,792.25 | 1.02 | 33.49 | 0.85 | |
| PuDDingRM Blocks=4(12.5%)2026.06 | — | — | 1,491.73 | 0.85 | 47.7 | 1.21 | |
| PuDDingRM Blocks=8(25%)2026.06 | — | — | 1,691.74 | 0.96 | 55.96 | 1.42 | |
| PuDDingRM Blocks=12(37.5%)2026.06 | — | — | 1,837.07 | 1.05 | 62.82 | 1.59 | |
| PuDDingRM Blocks=16(50%)2026.06 | — | — | 2,196.42 | 1.25 | 81.63 | 2.07 |