Code Generation on HumanEval (Speedup, Average Accepted Length, Drafting Cost)
3.92SpeedupSpecBlock
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| SpecBlockModel=Llama-3.1-8B, Temperature=02026.05 | 3.92 | 5.16 | 16 | |
| EAGLE-3Model=Llama-3.1-8B, Temperature=02026.05 | 3.59 | 6.98 | 31 | |
| SpecBlockModel=Llama-3.1-8B, Temperature=1.02026.05 | 3.21 | 4.62 | 14 | |
| FalconModel=Llama-3.1-8B, Temperature=02026.05 | 3.06 | 4.69 | 26 | |
| EAGLE-3Model=Llama-3.1-8B, Temperature=1.02026.05 | 2.98 | 6.09 | 30 | |
| FalconModel=Llama-3.1-8B, Temperature=1.02026.05 | 2.64 | 3.86 | 27 | |
| SpecBlockModel=Qwen3-32B, Temperature=02026.05 | 2.61 | 3.47 | 11 | |
| EAGLE-3Model=Qwen3-32B, Temperature=02026.05 | 2.54 | 4.36 | 24 | |
| ParallelSpecModel=Llama-3.1-8B, Temperature=02026.05 | 2.5 | 3.16 | 8 | |
| SpecBlockModel=Qwen3-8B, Temperature=02026.05 | 2.5 | 3.59 | 14 | |
| EAGLE-3Model=Qwen3-8B, Temperature=02026.05 | 2.45 | 4.53 | 29 | |
| SpecBlockModel=Qwen3-8B, Temperature=1.02026.05 | 2.44 | 3.53 | 14 | |
| EAGLE-3Model=Qwen3-8B, Temperature=1.02026.05 | 2.32 | 4.34 | 29 | |
| ParallelSpecModel=Llama-3.1-8B, Temperature=1.02026.05 | 2.27 | 2.65 | 9 | |
| SpecBlockModel=Qwen3-32B, Temperature=1.02026.05 | 2.26 | 3.43 | 11 | |
| EAGLE-3Model=Qwen3-32B, Temperature=1.02026.05 | 2.17 | 4.21 | 25 | |
| MedusaModel=Llama-3.1-8B, Temperature=02026.05 | 2.16 | 2.7 | 6 | |
| MedusaModel=Llama-3.1-8B, Temperature=1.02026.05 | 1.85 | 2.2 | 7 | |
| SpSModel=Llama-3.1-8B, Temperature=02026.05 | 1.55 | 2.91 | 38 | |
| SpSModel=Llama-3.1-8B, Temperature=1.02026.05 | 1.33 | 2.32 | 40 |