Speculative Decoding on gsm8k
9.43Average Generation Length (τ)TreeFlash
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| TreeFlashTarget model=Qwen3 8B, Temperature (T)=0, Draft budget (B)=642026.06 | 9.43 | — | 6.73 | |
| TreeFlashTarget model=Qwen3 4B, Temperature (T)=0, Draft budget (B)=642026.06 | 9.39 | — | 6.7 | |
| DDTreeHardware=NVIDIA A800-SXM4-80GB GPU, Target Model=Qwen3-8B2026.05 | 9.27 | — | 3.97 | |
| DDTreeHardware=NVIDIA A800-SXM4-80GB GPU, Target Model=Qwen3-4B2026.05 | 9.26 | — | 5.87 | |
| DDTreeHardware=NVIDIA A40-48GB GPU, Target Model=Qwen3-8B2026.05 | 9.25 | — | 2.84 | |
| DDTreeHardware=NVIDIA A40-48GB GPU, Target Model=Qwen3-4B2026.05 | 9.14 | — | 3.57 | |
| TreeFlashTarget model=Qwen3 4B, Temperature (T)=1, Draft budget (B)=642026.06 | 8.72 | — | 6.22 | |
| TreeFlashTarget model=Qwen3 8B, Temperature (T)=1, Draft budget (B)=642026.06 | 8.69 | — | 6.21 | |
| TAPSHardware=NVIDIA A40-48GB GPU, Target Model=Qwen3-4B2026.05 | 8.26 | — | 6.98 | |
| DDTreeTarget model=Qwen3 8B, Temperature (T)=0, Draft budget (B)=642026.06 | 8.25 | — | 6.15 | |
| DDTreeTarget model=Qwen3 4B, Temperature (T)=0, Draft budget (B)=642026.06 | 8.24 | — | 6.07 | |
| TAPSHardware=NVIDIA A800-SXM4-80GB GPU, Target Model=Qwen3-4B2026.05 | 8.23 | — | 6.62 | |
| TAPSHardware=NVIDIA A800-SXM4-80GB GPU, Target Model=Qwen3-8B2026.05 | 8.17 | — | 6.55 | |
| TAPSHardware=NVIDIA A40-48GB GPU, Target Model=Qwen3-8B2026.05 | 8.11 | — | 6.57 | |
| DDTreeTarget model=Qwen3 4B, Temperature (T)=1, Draft budget (B)=642026.06 | 7.74 | — | 5.71 | |
| TreeFlashTarget model=Qwen3 8B, Temperature (T)=0, Draft budget (B)=162026.06 | 7.71 | — | 5.33 | |
| DDTreeTarget model=Qwen3 8B, Temperature (T)=1, Draft budget (B)=642026.06 | 7.67 | — | 5.67 | |
| TreeFlashTarget model=Qwen3 4B, Temperature (T)=0, Draft budget (B)=162026.06 | 7.62 | — | 5.29 | |
| TreeFlashTarget model=Qwen3 4B, Temperature (T)=1, Draft budget (B)=162026.06 | 7.15 | — | 4.98 | |
| DDTreeTarget model=Qwen3 8B, Temperature (T)=0, Draft budget (B)=162026.06 | 7.14 | — | 5.15 | |
| DDTreeTarget model=Qwen3 4B, Temperature (T)=0, Draft budget (B)=162026.06 | 7.13 | — | 5.13 | |
| TreeFlashTarget model=Qwen3 8B, Temperature (T)=1, Draft budget (B)=162026.06 | 7.03 | — | 4.85 | |
| DDTreeTarget model=Qwen3 4B, Temperature (T)=1, Draft budget (B)=162026.06 | 6.66 | — | 4.78 | |
| DDTreeTarget model=Qwen3 8B, Temperature (T)=1, Draft budget (B)=162026.06 | 6.62 | — | 4.77 | |
| DDTreeHardware=NVIDIA A40-48GB GPU, Target Model=LLaMA-3.1-8B-Instruct2026.05 | 6.53 | — | 1.66 | |
| DDTreeHardware=NVIDIA A800-SXM4-80GB GPU, Target Model=LLaMA-3.1-8B-Instruct2026.05 | 6.53 | — | 2.41 | |
| DFlashTarget model=Qwen3 8B, Temperature (T)=0, Draft budget (B)=162026.06 | 6.33 | — | 4.71 | |
| DFlashHardware=NVIDIA A40-48GB GPU, Target Model=Qwen3-4B2026.05 | 6.29 | — | 5.15 | |
| DFlashTarget model=Qwen3 4B, Temperature (T)=0, Draft budget (B)=162026.06 | 6.26 | — | 4.63 | |
| DFlashHardware=NVIDIA A40-48GB GPU, Target Model=Qwen3-8B2026.05 | 6.12 | — | 5.15 | |
| DFlashHardware=NVIDIA A800-SXM4-80GB GPU, Target Model=Qwen3-4B2026.05 | 6 | — | 4.71 | |
| DFlashHardware=NVIDIA A800-SXM4-80GB GPU, Target Model=Qwen3-8B2026.05 | 5.98 | — | 4.67 | |
| TAPSHardware=NVIDIA A800-SXM4-80GB GPU, Target Model=LLaMA-3.1-8B-Instruct2026.05 | 5.82 | — | 4.38 | |
| TAPSHardware=NVIDIA A40-48GB GPU, Target Model=LLaMA-3.1-8B-Instruct2026.05 | 5.78 | — | 4 | |
| MTP Lλ LKTarget Model=DeepSeek-V3-0324 (685B), Temperature=02026.02 | 5.72 | — | — | |
| MTP KLTarget Model=DeepSeek-V3-0324 (685B), Temperature=02026.02 | 5.67 | — | — | |
| DFlashTarget model=Qwen3 4B, Temperature (T)=1, Draft budget (B)=162026.06 | 5.65 | — | 4.19 | |
| DFlashTarget model=Qwen3 8B, Temperature (T)=1, Draft budget (B)=162026.06 | 5.65 | — | 4.21 | |
| HASSModel Variant=Mixed 70k+70k, Temperature=02026.03 | 5.53 | — | — | |
| MTP Lλ LKTarget Model=DeepSeek-V3-0324 (685B), Temperature=12026.02 | 5.51 | — | — | |
| Griffin+GSBackbone=Meta-Llama-3-8B-Instruct, Decoding Strategy=greedy decoding2026.03 | 5.47 | — | 3.19 | |
| Griffin+LTDBackbone=Meta-Llama-3-8B-Instruct, Decoding Strategy=greedy decoding2026.03 | 5.45 | — | 3.13 | |
| HASSModel Variant=Merged Trees, Temperature=02026.03 | 5.42 | — | — | |
| EAGLE-2Model Variant=Merged Trees, Temperature=02026.03 | 5.32 | — | — | |
| MTP KLTarget Model=DeepSeek-V3-0324 (685B), Temperature=12026.02 | 5.23 | — | — | |
| HASSModel Variant=Merged Trees, Temperature=12026.03 | 5.21 | — | — | |
| EAGLE-3 Lλ LKTarget Model=LLaMA 3.3 70B Instruct, Temperature=02026.02 | 5.21 | — | — | |
| GriffinBackbone=Meta-Llama-3-8B-Instruct, Decoding Strategy=greedy decoding2026.03 | 5.16 | — | 3.18 | |
| EAGLE-3 KLTarget Model=LLaMA 3.3 70B Instruct, Temperature=02026.02 | 5.16 | — | — | |
| EAGLE-2Model Variant=MathInstruct, Temperature=02026.03 | 5.04 | — | — | |
| HASSModel Variant=MathInstruct, Temperature=02026.03 | 5.02 | — | — | |
| HASSModel Variant=Confidence Routed, Temperature=02026.03 | 5.01 | — | — | |
| EAGLE-2Model Variant=Merged Trees, Temperature=12026.03 | 5.01 | — | — | |
| EAGLE-3 Lλ LKTarget Model=LLaMA 3.3 70B Instruct, Temperature=12026.02 | 5.01 | — | — | |
| EAGLE-2Model Variant=Confidence Routed, Temperature=02026.03 | 4.91 | — | — | |
| EAGLE-3 KLTarget Model=LLaMA 3.3 70B Instruct, Temperature=12026.02 | 4.89 | — | — | |
| EAGLE-3 Lλ LKTarget Model=Qwen3-235B-A22B-Instruct-2507, Temperature=02026.02 | 4.87 | — | — | |
| HASSModel Variant=Mixed 35k+35k, Temperature=02026.03 | 4.77 | — | — | |
| EAGLE-3 KLTarget Model=Qwen3-235B-A22B-Instruct-2507, Temperature=02026.02 | 4.76 | — | — | |
| HASSModel Variant=MathInstruct, Temperature=12026.03 | 4.75 | — | — | |
| HASSModel Variant=Confidence Routed, Temperature=12026.03 | 4.72 | — | — | |
| EAGLE-2Model Variant=MathInstruct, Temperature=12026.03 | 4.71 | — | — | |
| EAGLE-2Model Variant=Mixed 70k+70k, Temperature=02026.03 | 4.68 | — | — | |
| HASSModel Variant=Mixed 35k+35k, Temperature=12026.03 | 4.66 | — | — | |
| EAGLE-2Model Variant=Confidence Routed, Temperature=12026.03 | 4.65 | — | — | |
| EAGLE-3 Lλ LKTarget Model=Qwen3-235B-A22B-Instruct-2507, Temperature=12026.02 | 4.65 | — | — | |
| EAGLE-3 Lλ LKTarget Model=LLaMA 3.1 8B Instruct, Temperature=02026.02 | 4.57 | — | — | |
| EAGLE-3 KLTarget Model=LLaMA 3.1 8B Instruct, Temperature=02026.02 | 4.5 | — | — | |
| DFlashHardware=NVIDIA A800-SXM4-80GB GPU, Target Model=LLaMA-3.1-8B-Instruct2026.05 | 4.27 | — | 3.09 | |
| EAGLE-3 KLTarget Model=Qwen3-235B-A22B-Instruct-2507, Temperature=12026.02 | 4.27 | — | — | |
| DFlashHardware=NVIDIA A40-48GB GPU, Target Model=LLaMA-3.1-8B-Instruct2026.05 | 4.25 | — | 3.06 | |
| HASSModel Variant=Mixed 70k+70k, Temperature=12026.03 | 4.16 | — | — | |
| EAGLE-2Model Variant=Mixed 35k+35k, Temperature=02026.03 | 4.12 | — | — | |
| HASSModel Variant=ShareGPT, Temperature=02026.03 | 4.09 | — | — | |
| EAGLE-2Model Variant=Mixed 35k+35k, Temperature=12026.03 | 4.08 | — | — | |
| HASSModel Variant=ShareGPT, Temperature=12026.03 | 4.03 | — | — | |
| EAGLE-3 Lλ LKTarget Model=LLaMA 3.1 8B Instruct, Temperature=12026.02 | 4.02 | — | — | |
| EAGLE-3 KLTarget Model=GPT-OSS 20B, Temperature=02026.02 | 4.01 | — | — | |
| EAGLE-3 Lλ LKTarget Model=GPT-OSS 20B, Temperature=02026.02 | 4 | — | — | |
| EAGLE-3 KLTarget Model=LLaMA 3.1 8B Instruct, Temperature=12026.02 | 3.88 | — | — | |
| EAGLE-2Model Variant=Mixed 70k+70k, Temperature=12026.03 | 3.76 | — | — | |
| EAGLE3Target model=Qwen3 8B, Temperature (T)=0, Draft budget (B)=642026.06 | 3.75 | — | 2.17 | |
| EAGLE3Target model=Qwen3 4B, Temperature (T)=0, Draft budget (B)=642026.06 | 3.73 | — | 2.14 | |
| EAGLE-2Model Variant=ShareGPT, Temperature=02026.03 | 3.72 | — | — | |
| EAGLE-2Model Variant=ShareGPT, Temperature=12026.03 | 3.72 | — | — | |
| EAGLE3Target model=Qwen3 4B, Temperature (T)=1, Draft budget (B)=642026.06 | 3.66 | — | 2.06 | |
| EAGLE-3 Lλ LKTarget Model=GPT-OSS 20B, Temperature=12026.02 | 3.65 | — | — | |
| EAGLE3Target model=Qwen3 8B, Temperature (T)=1, Draft budget (B)=642026.06 | 3.63 | — | 2.06 | |
| EAGLE-3 KLTarget Model=GPT-OSS 20B, Temperature=12026.02 | 3.51 | — | — | |
| MTP originalTarget Model=DeepSeek-V3-0324 (685B), Temperature=02026.02 | 3.41 | — | — | |
| EAGLE-3Hardware=NVIDIA A40-48GB GPU, Target Model=Qwen3-4B2026.05 | 3.3 | — | 1.99 | |
| MTP originalTarget Model=DeepSeek-V3-0324 (685B), Temperature=12026.02 | 3.27 | — | — | |
| EAGLE-3Hardware=NVIDIA A40-48GB GPU, Target Model=Qwen3-8B2026.05 | 3.23 | — | 1.94 | |
| EAGLE-3Hardware=NVIDIA A800-SXM4-80GB GPU, Target Model=Qwen3-4B2026.05 | 3.22 | — | 1.89 | |
| EAGLE3Target model=Qwen3 8B, Temperature (T)=0, Draft budget (B)=162026.06 | 3.21 | — | 1.88 | |
| EAGLE3Target model=Qwen3 4B, Temperature (T)=0, Draft budget (B)=162026.06 | 3.18 | — | 1.84 | |
| EAGLE3Target model=Qwen3 4B, Temperature (T)=1, Draft budget (B)=162026.06 | 3.15 | — | 1.79 | |
| EAGLE3Target model=Qwen3 8B, Temperature (T)=1, Draft budget (B)=162026.06 | 3.13 | — | 1.82 | |
| EAGLE-3Hardware=NVIDIA A800-SXM4-80GB GPU, Target Model=Qwen3-8B2026.05 | 3.12 | — | 1.87 | |
| EAGLE-3 Lλ LKTarget Model=GPT-OSS 120B, Temperature=02026.02 | 3.06 | — | — |