Code Generation on HumanEval (Acceptance Length and Speedup)
7.07Average Acceptance LengthBlockPilot
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| BlockPilotModel=Qwen3-8B, Temperature=02026.06 | 7.07 | 5.21 | — | |
| BlockPilotModel=Qwen3-4B, Temperature=02026.06 | 6.77 | 4.46 | — | |
| DFlashModel=Qwen3-8B, Temperature=0, Block size (n)=162026.06 | 6.76 | 4.98 | — | |
| DFlashModel=Qwen3-4B, Temperature=0, Block size (n)=162026.06 | 6.51 | 4.28 | — | |
| dFlashModel=Qwen3-8B, Scaled training=true2026.05 | 6.47 | 4.68 | — | |
| BlockPilotModel=Qwen3-4B, Temperature=12026.06 | 6.45 | 4.74 | — | |
| DFlashModel=Qwen3-8B, Temperature=0, Block size (n)=322026.06 | 6.32 | 4.66 | — | |
| BlockPilotModel=Qwen3-8B, Temperature=12026.06 | 6.3 | 4.65 | — | |
| DFlashModel=Qwen3-4B, Temperature=1, Block size (n)=162026.06 | 5.87 | 4.32 | — | |
| DFlashModel=Qwen3-8B, Temperature=1, Block size (n)=162026.06 | 5.68 | 4.19 | — | |
| DSparktarget_model=Gemma4-12B2026.07 | 5.64 | — | — | |
| FlexDraftModel=Qwen3-8B, Scaled training=true2026.05 | 5.55 | 4.26 | — | |
| DSparktarget_model=Qwen3-8B2026.07 | 5.52 | — | — | |
| DSparktarget_model=Qwen3-14B2026.07 | 5.43 | — | — | |
| DSparktarget_model=Qwen3-4B2026.07 | 5.38 | — | — | |
| Eagle3target_model=Gemma4-12B2026.07 | 5.37 | — | — | |
| DFlashModel=Qwen3-8B, Temperature=0, Block size (n)=82026.06 | 5.19 | 3.82 | — | |
| DFlashModel=Qwen3-8B, Temperature=1, Block size (n)=322026.06 | 5.11 | 3.77 | — | |
| DFlashtarget_model=Gemma4-12B2026.07 | 4.95 | — | — | |
| DFlashModel=Qwen3-4B, Temperature=0, Block size (n)=82026.06 | 4.84 | 3.19 | — | |
| DFlashtarget_model=Qwen3-4B2026.07 | 4.74 | — | — | |
| DFlashtarget_model=Qwen3-8B2026.07 | 4.64 | — | — | |
| DFlashtarget_model=Qwen3-14B2026.07 | 4.59 | — | — | |
| DFlashModel=Qwen3-4B, Temperature=1, Block size (n)=82026.06 | 4.58 | 3.37 | — | |
| DFlashModel=Qwen3-8B, Temperature=1, Block size (n)=82026.06 | 4.41 | 3.26 | — | |
| Eagle3target_model=Qwen3-8B2026.07 | 4.33 | — | — | |
| Eagle3target_model=Qwen3-4B2026.07 | 4.16 | — | — | |
| Eagle3target_model=Qwen3-14B2026.07 | 4.14 | — | — | |
| DFlashModel=Qwen3-4B, Temperature=0, Block size (n)=322026.06 | 3.89 | 2.56 | — | |
| DFlashModel=Qwen3-4B, Temperature=1, Block size (n)=322026.06 | 3.76 | 2.76 | — | |
| EAGLE-3Model=Qwen3-8B, Scaled training=true2026.05 | 3.74 | 2.52 | — | |
| DFlashModel=Qwen3-8B, Temperature=0, Block size (n)=42026.06 | 3.33 | 2.45 | — | |
| DFlashModel=Qwen3-4B, Temperature=0, Block size (n)=42026.06 | 3.22 | 2.12 | — | |
| EAGLE-3Model=Qwen3-8B, Temperature=02026.06 | 3.17 | 1.89 | — | |
| DFlashModel=Qwen3-4B, Temperature=1, Block size (n)=42026.06 | 3.11 | 2.29 | — | |
| EAGLE-3Model=Qwen3-4B, Temperature=02026.06 | 3.05 | 1.84 | — | |
| EAGLE-3Model=Qwen3-8B, Temperature=12026.06 | 3.05 | 1.75 | — | |
| DFlashModel=Qwen3-8B, Temperature=1, Block size (n)=42026.06 | 3.05 | 2.25 | — | |
| EAGLE-3Model=Qwen3-4B, Temperature=12026.06 | 3.01 | 1.74 | — | |
| DARTModel=Qwen3-8B, Scaled training=true2026.05 | 2.95 | 2.52 | — | |
| Apple MTPModel=Qwen3-8B, Scaled training=true2026.05 | 2.49 | 2.09 | — | |
| BlockPilotModel=Qwen2.5-Coder-32B-Instruct, Temperature=02026.06 | — | 6.05 | 9.07 | |
| BlockPilotModel=Qwen2.5-Coder-32B-Instruct, Temperature=12026.06 | — | 6.35 | 9.39 |