Speculative Decoding on MT-bench
6.06Tau (τ)DDTree
Evaluation results
| Method | Links | ||
|---|---|---|---|
| DDTreeHardware=NVIDIA A40-48GB GPU, Target Model=LLaMA-3.1-8B-Instruct2026.05 | 6.06 | 1.49 | |
| DDTreeHardware=NVIDIA A800-SXM4-80GB GPU, Target Model=LLaMA-3.1-8B-Instruct2026.05 | 6.05 | 2.17 | |
| DDTreeHardware=NVIDIA A800-SXM4-80GB GPU, Target Model=Qwen3-4B2026.05 | 5.79 | 3.53 | |
| DDTreeHardware=NVIDIA A800-SXM4-80GB GPU, Target Model=Qwen3-8B2026.05 | 5.68 | 2.26 | |
| TreeFlashTarget model=Qwen3 8B, Temperature (T)=0, Draft budget (B)=642026.06 | 5.42 | 4 | |
| DDTreeHardware=NVIDIA A40-48GB GPU, Target Model=Qwen3-4B2026.05 | 5.41 | 2.32 | |
| DDTreeHardware=NVIDIA A40-48GB GPU, Target Model=Qwen3-8B2026.05 | 5.4 | 1.71 | |
| TreeFlashTarget model=Qwen3 4B, Temperature (T)=0, Draft budget (B)=642026.06 | 5.38 | 3.92 | |
| TAPSHardware=NVIDIA A40-48GB GPU, Target Model=LLaMA-3.1-8B-Instruct2026.05 | 5.36 | 3.61 | |
| TAPSHardware=NVIDIA A800-SXM4-80GB GPU, Target Model=LLaMA-3.1-8B-Instruct2026.05 | 5.3 | 3.85 | |
| Griffin+GSBackbone=Meta-Llama-3-8B-Instruct, Decoding Strategy=greedy decoding2026.03 | 5.11 | 2.81 | |
| TAPSHardware=NVIDIA A800-SXM4-80GB GPU, Target Model=Qwen3-4B2026.05 | 5 | 4.04 | |
| Griffin+LTDBackbone=Meta-Llama-3-8B-Instruct, Decoding Strategy=greedy decoding2026.03 | 4.98 | 2.88 | |
| TreeFlashTarget model=Qwen3 4B, Temperature (T)=1, Draft budget (B)=642026.06 | 4.92 | 3.59 | |
| DDTreeTarget model=Qwen3 4B, Temperature (T)=0, Draft budget (B)=642026.06 | 4.9 | 3.68 | |
| DDTreeTarget model=Qwen3 8B, Temperature (T)=0, Draft budget (B)=642026.06 | 4.89 | 3.73 | |
| TAPSHardware=NVIDIA A800-SXM4-80GB GPU, Target Model=Qwen3-8B2026.05 | 4.87 | 3.65 | |
| GriffinBackbone=Meta-Llama-3-8B-Instruct, Decoding Strategy=greedy decoding2026.03 | 4.82 | 1.83 | |
| TreeFlashTarget model=Qwen3 8B, Temperature (T)=1, Draft budget (B)=642026.06 | 4.7 | 3.46 | |
| TAPSHardware=NVIDIA A40-48GB GPU, Target Model=Qwen3-8B2026.05 | 4.67 | 3.92 | |
| TAPSHardware=NVIDIA A40-48GB GPU, Target Model=Qwen3-4B2026.05 | 4.58 | 4.26 | |
| TreeFlashTarget model=Qwen3 4B, Temperature (T)=0, Draft budget (B)=162026.06 | 4.57 | 3.23 | |
| TreeFlashTarget model=Qwen3 8B, Temperature (T)=0, Draft budget (B)=162026.06 | 4.53 | 3.21 | |
| DDTreeTarget model=Qwen3 4B, Temperature (T)=1, Draft budget (B)=642026.06 | 4.47 | 3.36 | |
| DDTreeTarget model=Qwen3 8B, Temperature (T)=1, Draft budget (B)=642026.06 | 4.28 | 3.22 | |
| DDTreeTarget model=Qwen3 4B, Temperature (T)=0, Draft budget (B)=162026.06 | 4.24 | 3.09 | |
| DDTreeTarget model=Qwen3 8B, Temperature (T)=0, Draft budget (B)=162026.06 | 4.14 | 3.05 | |
| TreeFlashTarget model=Qwen3 4B, Temperature (T)=1, Draft budget (B)=162026.06 | 4.08 | 2.88 | |
| DFlashHardware=NVIDIA A800-SXM4-80GB GPU, Target Model=Qwen3-4B2026.05 | 4.07 | 2.67 | |
| EAGLE-3 KLTarget Model=LLaMA 3.3 70B Instruct, Temperature=02026.02 | 4.01 | — | |
| EAGLE-3 Lλ LKTarget Model=LLaMA 3.3 70B Instruct, Temperature=02026.02 | 4 | — | |
| MTP Lλ LKTarget Model=DeepSeek-V3-0324 (685B), Temperature=02026.02 | 4 | — | |
| MTP KLTarget Model=DeepSeek-V3-0324 (685B), Temperature=02026.02 | 3.96 | — | |
| TreeFlashTarget model=Qwen3 8B, Temperature (T)=1, Draft budget (B)=162026.06 | 3.92 | 2.76 | |
| EAGLE-3 Lλ LKTarget Model=LLaMA 3.3 70B Instruct, Temperature=12026.02 | 3.89 | — | |
| DDTreeTarget model=Qwen3 4B, Temperature (T)=1, Draft budget (B)=162026.06 | 3.89 | 2.84 | |
| MTP Lλ LKTarget Model=DeepSeek-V3-0324 (685B), Temperature=12026.02 | 3.88 | — | |
| EAGLE-3 Lλ LKTarget Model=LLaMA 3.1 8B Instruct, Temperature=02026.02 | 3.84 | — | |
| DFlashHardware=NVIDIA A40-48GB GPU, Target Model=LLaMA-3.1-8B-Instruct2026.05 | 3.82 | 2.74 | |
| DFlashHardware=NVIDIA A800-SXM4-80GB GPU, Target Model=Qwen3-8B2026.05 | 3.8 | 2.47 | |
| DFlashHardware=NVIDIA A800-SXM4-80GB GPU, Target Model=LLaMA-3.1-8B-Instruct2026.05 | 3.79 | 2.75 | |
| EAGLE-3 KLTarget Model=LLaMA 3.3 70B Instruct, Temperature=12026.02 | 3.76 | — | |
| EAGLE-3 KLTarget Model=LLaMA 3.1 8B Instruct, Temperature=02026.02 | 3.75 | — | |
| MTP KLTarget Model=DeepSeek-V3-0324 (685B), Temperature=12026.02 | 3.66 | — | |
| DDTreeTarget model=Qwen3 8B, Temperature (T)=1, Draft budget (B)=162026.06 | 3.65 | 2.68 | |
| EAGLE-3 Lλ LKTarget Model=LLaMA 3.1 8B Instruct, Temperature=12026.02 | 3.48 | — | |
| DFlashTarget model=Qwen3 4B, Temperature (T)=0, Draft budget (B)=162026.06 | 3.48 | 2.66 | |
| EAGLE3Target model=Qwen3 4B, Temperature (T)=0, Draft budget (B)=642026.06 | 3.41 | 1.96 | |
| EAGLE-3 KLTarget Model=LLaMA 3.1 8B Instruct, Temperature=12026.02 | 3.39 | — | |
| EAGLE-3 Lλ LKTarget Model=Qwen3-235B-A22B-Instruct-2507, Temperature=02026.02 | 3.36 | — | |
| EAGLE-3 Lλ LKTarget Model=GPT-OSS 20B, Temperature=02026.02 | 3.35 | — | |
| EAGLE-3 KLTarget Model=Qwen3-235B-A22B-Instruct-2507, Temperature=02026.02 | 3.33 | — | |
| DFlashTarget model=Qwen3 8B, Temperature (T)=0, Draft budget (B)=162026.06 | 3.32 | 2.55 | |
| EAGLE-3 KLTarget Model=GPT-OSS 20B, Temperature=02026.02 | 3.31 | — | |
| EAGLE3Target model=Qwen3 8B, Temperature (T)=0, Draft budget (B)=642026.06 | 3.3 | 1.93 | |
| EAGLE3Target model=Qwen3 4B, Temperature (T)=1, Draft budget (B)=642026.06 | 3.27 | 1.83 | |
| EAGLE-3 Lλ LKTarget Model=GPT-OSS 20B, Temperature=12026.02 | 3.2 | — | |
| EAGLE-3 Lλ LKTarget Model=Qwen3-235B-A22B-Instruct-2507, Temperature=12026.02 | 3.18 | — | |
| DFlashHardware=NVIDIA A40-48GB GPU, Target Model=Qwen3-8B2026.05 | 3.16 | 2.75 | |
| DFlashHardware=NVIDIA A40-48GB GPU, Target Model=Qwen3-4B2026.05 | 3.15 | 2.85 | |
| EAGLE-3 KLTarget Model=GPT-OSS 20B, Temperature=12026.02 | 3.12 | — | |
| DFlashTarget model=Qwen3 4B, Temperature (T)=1, Draft budget (B)=162026.06 | 3.08 | 2.34 | |
| EAGLE-3Hardware=NVIDIA A40-48GB GPU, Target Model=Qwen3-4B2026.05 | 3.02 | 1.74 | |
| EAGLE3Target model=Qwen3 8B, Temperature (T)=1, Draft budget (B)=642026.06 | 2.98 | 1.71 | |
| EAGLE3Target model=Qwen3 4B, Temperature (T)=0, Draft budget (B)=162026.06 | 2.97 | 1.71 | |
| EAGLE-3 KLTarget Model=Qwen3-235B-A22B-Instruct-2507, Temperature=12026.02 | 2.96 | — | |
| EAGLE-3Hardware=NVIDIA A800-SXM4-80GB GPU, Target Model=Qwen3-4B2026.05 | 2.95 | 1.7 | |
| MTP originalTarget Model=DeepSeek-V3-0324 (685B), Temperature=02026.02 | 2.9 | — | |
| DFlashTarget model=Qwen3 8B, Temperature (T)=1, Draft budget (B)=162026.06 | 2.89 | 2.19 | |
| EAGLE-3 Lλ LKTarget Model=GPT-OSS 120B, Temperature=02026.02 | 2.86 | — | |
| EAGLE3Target model=Qwen3 8B, Temperature (T)=0, Draft budget (B)=162026.06 | 2.86 | 1.69 | |
| EAGLE3Target model=Qwen3 4B, Temperature (T)=1, Draft budget (B)=162026.06 | 2.84 | 1.62 | |
| EAGLE-3Hardware=NVIDIA A40-48GB GPU, Target Model=Qwen3-8B2026.05 | 2.83 | 1.63 | |
| MTP originalTarget Model=DeepSeek-V3-0324 (685B), Temperature=12026.02 | 2.82 | — | |
| EAGLE-3 KLTarget Model=GPT-OSS 120B, Temperature=02026.02 | 2.81 | — | |
| EAGLE-3Hardware=NVIDIA A800-SXM4-80GB GPU, Target Model=Qwen3-8B2026.05 | 2.7 | 1.58 | |
| EAGLE-3 Lλ LKTarget Model=GPT-OSS 120B, Temperature=12026.02 | 2.69 | — | |
| EAGLE3Target model=Qwen3 8B, Temperature (T)=1, Draft budget (B)=162026.06 | 2.58 | 1.51 | |
| EAGLE-3Hardware=NVIDIA A40-48GB GPU, Target Model=LLaMA-3.1-8B-Instruct2026.05 | 2.57 | 1.85 | |
| EAGLE-3 KLTarget Model=GPT-OSS 120B, Temperature=12026.02 | 2.53 | — | |
| EAGLE-3Hardware=NVIDIA A800-SXM4-80GB GPU, Target Model=LLaMA-3.1-8B-Instruct2026.05 | 1.78 | 1.29 |