List your company
ResearchFormerly WizWand

Code Generation on HumanEval (Throughput, Speedup, and Avg. τ)

1.94Average Tau (τ)MEDUSA
1.58164.00086.428.8392Dec 17, 2025Dec 25, 2025Jan 2, 2026Jan 11, 2026Jan 19, 2026Jan 27, 2026Feb 5, 2026
Updated 8mo ago

Evaluation results

MethodLinks
MEDUSABase Model=Qwen3-4B, Temperature=0, KV Cache=true2025.121.94—1.34
MEDUSABase Model=Qwen3-8B, Temperature=0, KV Cache=true2025.121.97—1.32
HydraBase Model=Qwen3-4B, Temperature=0, KV Cache=true2025.122.53—2.3
EAGLE3Backbone=Qwen3-30B-A3B, temperature=0.6, KV cache=true2025.122.57—2.14
HydraBase Model=Qwen3-8B, Temperature=0, KV Cache=true2025.122.58—2.39
EAGLE3Backbone=Qwen3-14B, temperature=0.6, KV cache=true2025.122.61—1.91
EAGLE3Backbone=Qwen3-4B, temperature=0.6, KV cache=true2025.122.63—2.01
EAGLE-3Model=Q3-4B, Temperature=1, Draft tree size=162026.023.01—1.74
EAGLE-3Model=Q3-4B, Temperature=0, Draft tree size=162026.023.05—1.84
EAGLE-3Model=Q3-8B, Temperature=1, Draft tree size=162026.023.05—1.75
EAGLE-3Model=Q3-8B, Temperature=0, Draft tree size=162026.023.17—1.89
EAGLE3Base Model=Qwen3-30B-A3B, Temperature=0, KV Cache=true2025.123.21—2.41
EAGLE3Backbone=Qwen3-8B, temperature=0.6, KV cache=true2025.123.23—2.18
EAGLE3Backbone=Qwen2-7B, temperature=0.6, KV cache=true2025.123.25—2.41
EAGLE3Base Model=Qwen3-4B, Temperature=0, KV Cache=true2025.123.26—2.47
EAGLE3Base Model=Qwen2-7B, Temperature=0, KV Cache=true2025.123.34—2.65
EAGLE-3Model=Q3-4B, Temperature=1, Draft tree size=602026.023.39—1.94
EAGLE-3Model=Q3-4B, Temperature=0, Draft tree size=602026.023.47—2.12
EAGLE-3Model=Q3-8B, Temperature=1, Draft tree size=602026.023.54—2.05
EAGLE-3Model=Q3-8B, Temperature=0, Draft tree size=602026.023.65—2.17
EAGLE3Base Model=Qwen3-14B, Temperature=0, KV Cache=true2025.123.72—2.62
EAGLE3Base Model=Qwen3-8B, Temperature=0, KV Cache=true2025.123.87—2.65
DEERBackbone=Qwen2-7B, temperature=0.6, KV cache=true2025.123.9—2.52
DEERBase Model=Qwen2-7B, Temperature=0, KV Cache=true2025.124.21—2.79
DEERBackbone=Qwen3-4B, temperature=0.6, KV cache=true2025.124.51—2.59
AdaEDLBackbone Model=Llama-2 7B/70B2026.024.5722.53—
DEERBackbone=Qwen3-8B, temperature=0.6, KV cache=true2025.124.66—3.18
DEERBase Model=Qwen3-4B, Temperature=0, KV Cache=true2025.124.68—2.97
DEERBase Model=Qwen3-8B, Temperature=0, KV Cache=true2025.125—3.3
DEERBackbone=Qwen3-14B, temperature=0.6, KV cache=true2025.125—3.5
SpecDec++*Backbone Model=Llama-2 7B/70B, Note=Underperforms due to chat template differences2026.025.1321.17—
SpeculativeBackbone Model=Llama-2 7B/70B2026.025.3921.2—
DEERBackbone=Qwen3-30B-A3B, temperature=0.6, KV cache=true2025.125.48—4.32
DFlashModel=Q3-8B, Temperature=1, Diffusion block size=162026.025.52—4.32
DEERBase Model=Qwen3-14B, Temperature=0, KV Cache=true2025.125.72—3.59
DFlashModel=Q3-4B, Temperature=1, Diffusion block size=162026.026.04—4.74
DFlashModel=Qwen3-8B, Concurrency=1, Inference Engine=SGLang, Backend=FA42026.026.59554.2
DFlashModel=Q3-8B, Temperature=0, Diffusion block size=162026.026.5—5.14
DEERBase Model=Qwen3-30B-A3B, Temperature=0, KV Cache=true2025.126.58—5.54
DFlashModel=Qwen3-4B, Concurrency=1, Inference Engine=SGLang, Backend=FA42026.026.631,2474
DFlashModel=Q3-4B, Temperature=0, Diffusion block size=162026.026.64—5.21
PACERBackbone Model=Llama-2 7B/70B2026.027.4624.2—
DFlashModel=Qwen3-Coder-30B-A3B, Concurrency=1, Inference Engine=SGLang, Backend=FA42026.028.098023.5
OuroborosModels=Deepseek-Coder 1.3B/33B2026.028.3649.052.71
PACER + OuroborosModels=Deepseek-Coder 1.3B/33B2026.0210.951.072.82
BaselineModel=Qwen3-4B, Concurrency=1, Inference Engine=SGLang, Backend=FA42026.02—312—
BaselineModel=Qwen3-8B, Concurrency=1, Inference Engine=SGLang, Backend=FA42026.02—229—
BaselineModel=Qwen3-Coder-30B-A3B, Concurrency=1, Inference Engine=SGLang, Backend=FA42026.02—229—
VanillaModels=Deepseek-Coder 1.3B/33B2026.02—18.11