Code Generation on HumanEval (Throughput, Speedup, and Avg. τ)
1.94Average Tau (τ)MEDUSA
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| MEDUSABase Model=Qwen3-4B, Temperature=0, KV Cache=true2025.12 | 1.94 | — | 1.34 | |
| MEDUSABase Model=Qwen3-8B, Temperature=0, KV Cache=true2025.12 | 1.97 | — | 1.32 | |
| HydraBase Model=Qwen3-4B, Temperature=0, KV Cache=true2025.12 | 2.53 | — | 2.3 | |
| EAGLE3Backbone=Qwen3-30B-A3B, temperature=0.6, KV cache=true2025.12 | 2.57 | — | 2.14 | |
| HydraBase Model=Qwen3-8B, Temperature=0, KV Cache=true2025.12 | 2.58 | — | 2.39 | |
| EAGLE3Backbone=Qwen3-14B, temperature=0.6, KV cache=true2025.12 | 2.61 | — | 1.91 | |
| EAGLE3Backbone=Qwen3-4B, temperature=0.6, KV cache=true2025.12 | 2.63 | — | 2.01 | |
| EAGLE-3Model=Q3-4B, Temperature=1, Draft tree size=162026.02 | 3.01 | — | 1.74 | |
| EAGLE-3Model=Q3-4B, Temperature=0, Draft tree size=162026.02 | 3.05 | — | 1.84 | |
| EAGLE-3Model=Q3-8B, Temperature=1, Draft tree size=162026.02 | 3.05 | — | 1.75 | |
| EAGLE-3Model=Q3-8B, Temperature=0, Draft tree size=162026.02 | 3.17 | — | 1.89 | |
| EAGLE3Base Model=Qwen3-30B-A3B, Temperature=0, KV Cache=true2025.12 | 3.21 | — | 2.41 | |
| EAGLE3Backbone=Qwen3-8B, temperature=0.6, KV cache=true2025.12 | 3.23 | — | 2.18 | |
| EAGLE3Backbone=Qwen2-7B, temperature=0.6, KV cache=true2025.12 | 3.25 | — | 2.41 | |
| EAGLE3Base Model=Qwen3-4B, Temperature=0, KV Cache=true2025.12 | 3.26 | — | 2.47 | |
| EAGLE3Base Model=Qwen2-7B, Temperature=0, KV Cache=true2025.12 | 3.34 | — | 2.65 | |
| EAGLE-3Model=Q3-4B, Temperature=1, Draft tree size=602026.02 | 3.39 | — | 1.94 | |
| EAGLE-3Model=Q3-4B, Temperature=0, Draft tree size=602026.02 | 3.47 | — | 2.12 | |
| EAGLE-3Model=Q3-8B, Temperature=1, Draft tree size=602026.02 | 3.54 | — | 2.05 | |
| EAGLE-3Model=Q3-8B, Temperature=0, Draft tree size=602026.02 | 3.65 | — | 2.17 | |
| EAGLE3Base Model=Qwen3-14B, Temperature=0, KV Cache=true2025.12 | 3.72 | — | 2.62 | |
| EAGLE3Base Model=Qwen3-8B, Temperature=0, KV Cache=true2025.12 | 3.87 | — | 2.65 | |
| DEERBackbone=Qwen2-7B, temperature=0.6, KV cache=true2025.12 | 3.9 | — | 2.52 | |
| DEERBase Model=Qwen2-7B, Temperature=0, KV Cache=true2025.12 | 4.21 | — | 2.79 | |
| DEERBackbone=Qwen3-4B, temperature=0.6, KV cache=true2025.12 | 4.51 | — | 2.59 | |
| AdaEDLBackbone Model=Llama-2 7B/70B2026.02 | 4.57 | 22.53 | — | |
| DEERBackbone=Qwen3-8B, temperature=0.6, KV cache=true2025.12 | 4.66 | — | 3.18 | |
| DEERBase Model=Qwen3-4B, Temperature=0, KV Cache=true2025.12 | 4.68 | — | 2.97 | |
| DEERBase Model=Qwen3-8B, Temperature=0, KV Cache=true2025.12 | 5 | — | 3.3 | |
| DEERBackbone=Qwen3-14B, temperature=0.6, KV cache=true2025.12 | 5 | — | 3.5 | |
| SpecDec++*Backbone Model=Llama-2 7B/70B, Note=Underperforms due to chat template differences2026.02 | 5.13 | 21.17 | — | |
| SpeculativeBackbone Model=Llama-2 7B/70B2026.02 | 5.39 | 21.2 | — | |
| DEERBackbone=Qwen3-30B-A3B, temperature=0.6, KV cache=true2025.12 | 5.48 | — | 4.32 | |
| DFlashModel=Q3-8B, Temperature=1, Diffusion block size=162026.02 | 5.52 | — | 4.32 | |
| DEERBase Model=Qwen3-14B, Temperature=0, KV Cache=true2025.12 | 5.72 | — | 3.59 | |
| DFlashModel=Q3-4B, Temperature=1, Diffusion block size=162026.02 | 6.04 | — | 4.74 | |
| DFlashModel=Qwen3-8B, Concurrency=1, Inference Engine=SGLang, Backend=FA42026.02 | 6.5 | 955 | 4.2 | |
| DFlashModel=Q3-8B, Temperature=0, Diffusion block size=162026.02 | 6.5 | — | 5.14 | |
| DEERBase Model=Qwen3-30B-A3B, Temperature=0, KV Cache=true2025.12 | 6.58 | — | 5.54 | |
| DFlashModel=Qwen3-4B, Concurrency=1, Inference Engine=SGLang, Backend=FA42026.02 | 6.63 | 1,247 | 4 | |
| DFlashModel=Q3-4B, Temperature=0, Diffusion block size=162026.02 | 6.64 | — | 5.21 | |
| PACERBackbone Model=Llama-2 7B/70B2026.02 | 7.46 | 24.2 | — | |
| DFlashModel=Qwen3-Coder-30B-A3B, Concurrency=1, Inference Engine=SGLang, Backend=FA42026.02 | 8.09 | 802 | 3.5 | |
| OuroborosModels=Deepseek-Coder 1.3B/33B2026.02 | 8.36 | 49.05 | 2.71 | |
| PACER + OuroborosModels=Deepseek-Coder 1.3B/33B2026.02 | 10.9 | 51.07 | 2.82 | |
| BaselineModel=Qwen3-4B, Concurrency=1, Inference Engine=SGLang, Backend=FA42026.02 | — | 312 | — | |
| BaselineModel=Qwen3-8B, Concurrency=1, Inference Engine=SGLang, Backend=FA42026.02 | — | 229 | — | |
| BaselineModel=Qwen3-Coder-30B-A3B, Concurrency=1, Inference Engine=SGLang, Backend=FA42026.02 | — | 229 | — | |
| VanillaModels=Deepseek-Coder 1.3B/33B2026.02 | — | 18.1 | 1 |