Speculative Decoding on Spec-Bench
195.6MT ScoreFR-Spec
Evaluation results
| Method | Links | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| FR-SpecBase Model=Llama-3-8B, Temperature=0, Vocabulary Size (|Vhigh|)=32k, Implementation Framework=native C and CUDA2025.02 | 195.6 | — | 188.14 | 190.27 | 243.36 | 184.85 | — | 201.87 | — | 227.68 | 183.19 | 2.27 | — | |
| FR-SpecBase Model=Llama-3-8B, Temperature=0, Vocabulary Size (|Vhigh|)=16k, Implementation Framework=native C and CUDA2025.02 | 194.02 | — | 182.26 | 188.6 | 233.69 | 178.22 | — | 196.69 | — | 223.32 | 176.7 | 2.21 | — | |
| FR-SpecBase Model=Llama-3-8B, Temperature=0, Vocabulary Size (|Vhigh|)=64k, Implementation Framework=native C and CUDA2025.02 | 192.85 | — | 183.86 | 183.17 | 231.99 | 178.53 | — | 196.86 | — | 224.52 | 183.11 | 2.21 | — | |
| FR-SpecBase Model=Llama-3-8B, Temperature=0, Vocabulary Size (|Vhigh|)=8k, Implementation Framework=native C and CUDA2025.02 | 185.78 | — | 170.97 | 180.4 | 218.57 | 167.61 | — | 185.98 | — | 210.66 | 167.85 | 2.09 | — | |
| EAGLE-2Base Model=Llama-3-8B, Temperature=0, Implementation Framework=native C and CUDA2025.02 | 176.79 | — | 167.12 | 166.6 | 209.88 | 168.05 | — | 180.99 | — | 203.41 | 175.11 | 2.03 | — | |
| VanillaBase Model=Llama-3-8B, Temperature=0, Implementation Framework=native C and CUDA2025.02 | 90.94 | — | 86.63 | 91.05 | 91.16 | 83.43 | — | 89.11 | — | 90.43 | 90.1 | 1 | — | |
| PACERLLM Model=Llama-2 7B/70B2026.02 | 18.53 | 18.06 | 16.19 | 17.42 | 21.45 | 15.48 | — | — | 17.95 | — | — | — | — | |
| SpecDec++LLM Model=Llama-2 7B/70B2026.02 | 18.18 | 17.62 | 16.57 | 17.41 | 20.81 | 15.8 | — | — | 17.79 | — | — | — | — | |
| AssistLLM Model=Llama-2 7B/70B2026.02 | 17.77 | 17.95 | 16.42 | 17.1 | 20.46 | 15.6 | — | — | 17.58 | — | — | — | — | |
| AdaEDLLLM Model=Llama-2 7B/70B2026.02 | 17.55 | 17.85 | 16.1 | 17.34 | 20.21 | 15.69 | — | — | 17.47 | — | — | — | — | |
| VanillaLLM Model=Llama-2 7B/70B2026.02 | 9.04 | 9.31 | 8.63 | 9.32 | 9.29 | 8.32 | — | — | 8.99 | — | — | — | — | |
| SAMDTemperature=0.72025.12 | 4.62 | 3.12 | 4.18 | 3.38 | 4.16 | 3.95 | — | — | 3.9 | — | — | — | — | |
| Eagle-2Temperature=0.72025.12 | 4.54 | 3.19 | 3.81 | 3.41 | 4.38 | 3.8 | — | — | 3.86 | — | — | — | — | |
| O-PTPTemperature=0.7, Base Model=Vicuna-7B2025.12 | 4.52 | 3.73 | 4.54 | 3.42 | 4.62 | 4.29 | — | — | 4.18 | — | — | — | — | |
| HydraTemperature=0.72025.12 | 3.9 | 2.88 | 2.95 | 3.21 | 3.9 | 3.37 | — | — | 3.37 | — | — | — | — | |
| EAGLE-3 + SpecVocabTarget Model=Qwen3 8B2026.02 | 3.82 | — | 4.49 | 4.92 | 6.28 | 5.26 | — | — | 5.01 | 5.3 | — | — | — | |
| EAGLE-3 + FR-SpecTarget Model=Qwen3 8B2026.02 | 3.8 | — | 4.52 | 4.52 | 5.77 | 4.9 | — | — | 4.69 | 4.62 | — | — | — | |
| EAGLE-3 + VocabTrimTarget Model=Qwen3 8B2026.02 | 3.69 | — | 4.28 | 4.72 | 6.04 | 5.13 | — | — | 4.84 | 5.17 | — | — | — | |
| EagleTemperature=0.72025.12 | 3.66 | 2.74 | 3.15 | 2.81 | 3.57 | 3.21 | — | — | 3.19 | — | — | — | — | |
| EAGLE-3 (Our Reproduction)Target Model=Qwen3 8B, Protocol=Reproduction2026.02 | 3.6 | — | 4.21 | 4.8 | 5.94 | 5.04 | — | — | 4.78 | 5.11 | — | — | — | |
| EAGLE-3 + SpecVocab (Ours)Base Model=Qwen3 4B, Number of seeds=52026.02 | 3.59 | — | 4.63 | 5.09 | 6.05 | 5.45 | — | — | 5.01 | 5.26 | — | — | — | |
| EAGLE-3 + FR-SpecBase Model=Qwen3 4B, Number of seeds=52026.02 | 3.52 | — | 4.59 | 4.66 | 5.61 | 4.79 | — | — | 4.62 | 4.57 | — | — | — | |
| EAGLE-3 + SpecVocabBase Model=OLMo 2 1B2026.02 | 3.43 | — | 4.92 | 4.13 | 5.14 | 4.55 | — | — | 4.53 | 4.98 | — | — | — | |
| EAGLE-3 + VocabTrimBase Model=Qwen3 4B, Number of seeds=52026.02 | 3.43 | — | 4.35 | 5.05 | 5.86 | 5.04 | — | — | 4.82 | 5.2 | — | — | — | |
| EAGLE-3 + FR-SpecBase Model=OLMo 2 1B2026.02 | 3.41 | — | 4.77 | 4.04 | 4.87 | 4.32 | — | — | 4.33 | 4.56 | — | — | — | |
| EAGLE-3 (Our Reproduction)Base Model=Qwen3 4B, Number of seeds=52026.02 | 3.37 | — | 4.31 | 4.94 | 5.79 | 4.94 | — | — | 4.73 | 5.02 | — | — | — | |
| EAGLE-3Target Model=Qwen3 8B2026.02 | 3.34 | — | 3.86 | 4.46 | 5.28 | 4.52 | — | — | 4.33 | 4.52 | — | — | — | |
| EAGLE-3Vocab=128K Full Vocab2026.04 | 3.34 | — | 3.62 | 3.41 | 3.89 | 4.15 | — | — | 3.78 | 4.29 | — | — | — | |
| EvoSpecBase=Static 32k, Dynamic Expansion=Dyna Vocab 10242026.04 | 3.31 | — | 3.6 | 3.39 | 3.88 | 3.86 | — | — | 3.65 | 3.86 | — | — | — | |
| EvoSpecBase=Static 32k, Dynamic Expansion=Dyna Vocab 5122026.04 | 3.3 | — | 3.58 | 3.38 | 3.88 | 3.86 | — | — | 3.64 | 3.85 | — | — | — | |
| SAM-Decoding[E2]Model=Vicuna-13B2024.11 | 3.27 | 2.12 | 2.89 | 2.34 | 3.12 | 2.54 | 4.51 | 57.88 | 2.72 | — | — | — | — | |
| EAGLE-3 + VocabTrimBase Model=OLMo 2 1B2026.02 | 3.24 | — | 4.64 | 4.12 | 5.08 | 4.39 | — | — | 4.38 | 4.83 | — | — | — | |
| EAGLE-3Base Model=OLMo 2 1B, Reproduction=true2026.02 | 3.22 | — | 4.55 | 4 | 4.88 | 4.39 | — | — | 4.31 | 4.8 | — | — | — | |
| EvoSpecBase=Static 32k, Dynamic Expansion=Dyna Vocab 2562026.04 | 3.22 | — | 3.54 | 3.37 | 3.81 | 3.82 | — | — | 3.6 | 3.81 | — | — | — | |
| SAM-Decoding[E2]Model=Vicuna-7B, Hardware=A800 GPU, Draft Strategy=E22024.11 | 3.19 | 1.97 | 2.86 | 2.28 | 2.84 | 2.32 | 4.52 | 129.36 | 2.58 | — | — | — | — | |
| EAGLE-2Model=Vicuna-13B2024.11 | 3.1 | 2.15 | 2.58 | 2.38 | 3.19 | 2.33 | 4.42 | 56.06 | 2.63 | — | — | — | — | |
| FR-SpecVocab=Static 32k2026.04 | 3.1 | — | 3.51 | 3.29 | 3.79 | 3.81 | — | — | 3.54 | 3.73 | — | — | — | |
| RecyclingTemperature=0.72025.12 | 2.74 | 2.48 | 2.68 | 2.59 | 3.05 | 2.63 | — | — | 2.7 | — | — | — | — | |
| MedusaTemperature=0.72025.12 | 2.7 | 2.18 | 2.12 | 2.24 | 2.67 | 2.26 | — | — | 2.36 | — | — | — | — | |
| EAGLE-2Model=Vicuna-7B, Hardware=A800 GPU2024.11 | 2.66 | 1.76 | 2.18 | 2.03 | 2.63 | 1.97 | 4.34 | 110.56 | 2.21 | — | — | — | — | |
| SAM-Decoding[T]Model=Vicuna-7B, Hardware=A800 GPU, Draft Strategy=T2024.11 | 2.62 | 1.82 | 2.92 | 2.09 | 2.6 | 2.21 | 3.02 | 119.21 | 2.38 | — | — | — | — | |
| SAM-Decoding[T]Model=Vicuna-13B2024.11 | 2.36 | 1.8 | 2.63 | 1.83 | 2.49 | 2.22 | 2.91 | 47.27 | 2.22 | — | — | — | — | |
| SpSTemperature=0.72025.12 | 2.12 | 1.39 | 2.16 | 1.8 | 1.92 | 2.26 | — | — | 1.94 | — | — | — | — | |
| SAM-DecodingModel=Vicuna-13B2024.11 | 2.08 | 1.26 | 2.23 | 1.53 | 2.09 | 1.89 | 2.19 | 39.24 | 1.84 | — | — | — | — | |
| Token RecyclingModel=Vicuna-7B, Hardware=A800 GPU2024.11 | 2.08 | 1.76 | 1.97 | 1.85 | 2.35 | 1.76 | 2.82 | 98.39 | 1.96 | — | — | — | — | |
| Token RecyclingModel=Vicuna-13B2024.11 | 2.03 | 1.84 | 2.07 | 1.83 | 2.42 | 1.84 | 2.81 | 42.74 | 2.01 | — | — | — | — | |
| LogitSpecBackbone=Llama-3.1-8B-Instruct2025.07 | 1.89 | 1.67 | 1.94 | 1.68 | 2.01 | 1.77 | 2.11 | — | — | — | — | 1.88 | — | |
| LogitSpecBackbone=Qwen-3-8B2025.07 | 1.71 | 1.74 | 1.64 | 1.65 | 1.89 | 1.68 | 1.95 | — | — | — | — | 1.75 | — | |
| LookaheadTemperature=0.72025.12 | 1.69 | 1.24 | 1.54 | 1.56 | 1.92 | 1.42 | — | — | 1.56 | — | — | — | — | |
| PLDTemperature=0.72025.12 | 1.67 | 1.1 | 2.73 | 1.37 | 1.8 | 1.71 | — | — | 1.73 | — | — | — | — | |
| PLDModel=Vicuna-13B2024.11 | 1.61 | 1.1 | 2.36 | 1.11 | 1.69 | 1.8 | 1.66 | 33.89 | 1.59 | — | — | — | — | |
| NoneTemperature=0.72025.12 | 1 | 1 | 1 | 1 | 1 | 1 | — | — | 1 | — | — | — | — | |
| Autoregressive DecodingBase Model=OLMo 2 1B2026.02 | 1 | — | 1 | 1 | 1 | 1 | — | — | 1 | 1 | — | — | — | |
| Autoregressive DecodingTarget Model=Qwen3 8B2026.02 | 1 | — | 1 | 1 | 1 | 1 | — | — | 1 | 1 | — | — | — | |
| Autoregressive DecodingBase Model=Qwen3 4B, Number of seeds=52026.02 | 1 | — | 1 | 1 | 1 | 1 | — | — | 1 | 1 | — | — | — | |
| VanillaBackbone=Llama-3.1-8B-Instruct2025.07 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | — | — | — | — | 1 | — | |
| VanillaBackbone=Qwen-3-8B2025.07 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | — | — | — | — | 1 | — | |
| CactusVerifier model=Qwen 3 14B, Drafter model=Qwen 3 0.6B, Temperature=0.6, delta=12026.04 | — | — | — | — | — | — | — | — | — | — | — | 2.09 | 3.29 | |
| EAGLEBackbone=Vicuna-7B, Note=Requires training & extra heads2025.07 | — | — | — | — | — | — | 3.58 | — | — | — | — | 2.1 | — | |
| LogitSpecBackbone=Vicuna 7B2025.07 | — | — | — | — | — | — | 2.44 | — | — | — | — | 2.01 | — | |
| LogitSpecBackbone=Vicuna 13B2025.07 | — | — | — | — | — | — | 2.32 | — | — | — | — | 1.93 | — | |
| LogitSpecBackbone=Vicuna 33B2025.07 | — | — | — | — | — | — | 2.13 | — | — | — | — | 1.76 | — | |
| LogitSpecBackbone=Vicuna-7B, Note=Training-free & plug-and-play2025.07 | — | — | — | — | — | — | 2.44 | — | — | — | — | 2.01 | — | |
| LookaheadBackbone=Vicuna 7B2025.07 | — | — | — | — | — | — | 1.66 | — | — | — | — | 1.27 | — | |
| LookaheadBackbone=Vicuna 13B2025.07 | — | — | — | — | — | — | 1.63 | — | — | — | — | 1.22 | — | |
| LookaheadBackbone=Vicuna 33B2025.07 | — | — | — | — | — | — | 1.61 | — | — | — | — | 1.24 | — | |
| MedusaBackbone=Vicuna-7B, Note=Requires training & extra heads2025.07 | — | — | — | — | — | — | 2.31 | — | — | — | — | 1.71 | — | |
| PLDBackbone=Vicuna 7B2025.07 | — | — | — | — | — | — | 1.73 | — | — | — | — | 1.59 | — | |
| PLDBackbone=Vicuna 13B2025.07 | — | — | — | — | — | — | 1.68 | — | — | — | — | 1.48 | — | |
| PLDBackbone=Vicuna 33B2025.07 | — | — | — | — | — | — | 1.55 | — | — | — | — | 1.42 | — | |
| RESTBackbone=Vicuna 7B2025.07 | — | — | — | — | — | — | 1.82 | — | — | — | — | 1.48 | — | |
| RESTBackbone=Vicuna 13B2025.07 | — | — | — | — | — | — | 1.82 | — | — | — | — | 1.38 | — | |
| RESTBackbone=Vicuna 33B2025.07 | — | — | — | — | — | — | 1.8 | — | — | — | — | 1.48 | — | |
| SpSVerifier model=Qwen 3 14B, Drafter model=Qwen 3 0.6B, Temperature=0.62026.04 | — | — | — | — | — | — | — | — | — | — | — | 2.01 | 3.2 | |
| SpSBackbone=Vicuna 7B2025.07 | — | — | — | — | — | — | 2.28 | — | — | — | — | 1.49 | — | |
| SpSBackbone=Vicuna 13B2025.07 | — | — | — | — | — | — | 2.18 | — | — | — | — | 1.49 | — | |
| SpSBackbone=Vicuna 33B2025.07 | — | — | — | — | — | — | 2.01 | — | — | — | — | 1.61 | — |