List your company
ResearchFormerly WizWand

Code Generation on HumanEval (Acceptance Length and Speedup)

7.07Average Acceptance LengthBlockPilot
2.30683.54344.786.0166May 19, 2026May 27, 2026Jun 4, 2026Jun 12, 2026Jun 20, 2026Jun 28, 2026Jul 6, 2026
Updated 3mo ago

Evaluation results

MethodLinks
BlockPilotModel=Qwen3-8B, Temperature=02026.067.075.21—
BlockPilotModel=Qwen3-4B, Temperature=02026.066.774.46—
DFlashModel=Qwen3-8B, Temperature=0, Block size (n)=162026.066.764.98—
DFlashModel=Qwen3-4B, Temperature=0, Block size (n)=162026.066.514.28—
dFlashModel=Qwen3-8B, Scaled training=true2026.056.474.68—
BlockPilotModel=Qwen3-4B, Temperature=12026.066.454.74—
DFlashModel=Qwen3-8B, Temperature=0, Block size (n)=322026.066.324.66—
BlockPilotModel=Qwen3-8B, Temperature=12026.066.34.65—
DFlashModel=Qwen3-4B, Temperature=1, Block size (n)=162026.065.874.32—
DFlashModel=Qwen3-8B, Temperature=1, Block size (n)=162026.065.684.19—
DSparktarget_model=Gemma4-12B2026.075.64——
FlexDraftModel=Qwen3-8B, Scaled training=true2026.055.554.26—
DSparktarget_model=Qwen3-8B2026.075.52——
DSparktarget_model=Qwen3-14B2026.075.43——
DSparktarget_model=Qwen3-4B2026.075.38——
Eagle3target_model=Gemma4-12B2026.075.37——
DFlashModel=Qwen3-8B, Temperature=0, Block size (n)=82026.065.193.82—
DFlashModel=Qwen3-8B, Temperature=1, Block size (n)=322026.065.113.77—
DFlashtarget_model=Gemma4-12B2026.074.95——
DFlashModel=Qwen3-4B, Temperature=0, Block size (n)=82026.064.843.19—
DFlashtarget_model=Qwen3-4B2026.074.74——
DFlashtarget_model=Qwen3-8B2026.074.64——
DFlashtarget_model=Qwen3-14B2026.074.59——
DFlashModel=Qwen3-4B, Temperature=1, Block size (n)=82026.064.583.37—
DFlashModel=Qwen3-8B, Temperature=1, Block size (n)=82026.064.413.26—
Eagle3target_model=Qwen3-8B2026.074.33——
Eagle3target_model=Qwen3-4B2026.074.16——
Eagle3target_model=Qwen3-14B2026.074.14——
DFlashModel=Qwen3-4B, Temperature=0, Block size (n)=322026.063.892.56—
DFlashModel=Qwen3-4B, Temperature=1, Block size (n)=322026.063.762.76—
EAGLE-3Model=Qwen3-8B, Scaled training=true2026.053.742.52—
DFlashModel=Qwen3-8B, Temperature=0, Block size (n)=42026.063.332.45—
DFlashModel=Qwen3-4B, Temperature=0, Block size (n)=42026.063.222.12—
EAGLE-3Model=Qwen3-8B, Temperature=02026.063.171.89—
DFlashModel=Qwen3-4B, Temperature=1, Block size (n)=42026.063.112.29—
EAGLE-3Model=Qwen3-4B, Temperature=02026.063.051.84—
EAGLE-3Model=Qwen3-8B, Temperature=12026.063.051.75—
DFlashModel=Qwen3-8B, Temperature=1, Block size (n)=42026.063.052.25—
EAGLE-3Model=Qwen3-4B, Temperature=12026.063.011.74—
DARTModel=Qwen3-8B, Scaled training=true2026.052.952.52—
Apple MTPModel=Qwen3-8B, Scaled training=true2026.052.492.09—
BlockPilotModel=Qwen2.5-Coder-32B-Instruct, Temperature=02026.06—6.059.07
BlockPilotModel=Qwen2.5-Coder-32B-Instruct, Temperature=12026.06—6.359.39