Code Generation on HumanEval (MAT, Speed)
9.48MATTALON
Evaluation results
| Method | Links | ||
|---|---|---|---|
| TALONModel=Vicuna-13B, Temperature=02026.01 | 9.48 | 5.16 | |
| EAGLE-3Model=Vicuna-13B, Temperature=02026.01 | 8.31 | 4.77 | |
| OPT-TreeModel=Vicuna-13B, Temperature=02026.01 | 8.21 | 4.35 | |
| TALONModel=Llama3-8B, Temperature=02026.01 | 7.28 | 4.2 | |
| EAGLE-3Model=Llama3-8B, Temperature=02026.01 | 7.08 | 4.05 | |
| EAGLE-3Model=DSL-8B, Temperature=02026.01 | 6.7 | 3.85 | |
| TALONModel=DSL-8B, Temperature=02026.01 | 6.28 | 3.96 | |
| EAGLE-3Model=Qwen3-8B, Temperature=02026.01 | 3.89 | 2.35 | |
| HYDRAModel=Vicuna-13B, Temperature=02026.01 | 3.87 | 2.67 | |
| TALONModel=Qwen3-8B, Temperature=02026.01 | 3.78 | 2.69 | |
| EAGLE-3Model=Qwen3-32B, Temperature=02026.01 | 2.98 | 1.96 | |
| TALONModel=Qwen3-32B, Temperature=02026.01 | 2.96 | 2.15 | |
| SDModel=Vicuna-13B, Temperature=02026.01 | 2.86 | 1.47 | |
| MEDUSAModel=Vicuna-13B, Temperature=02026.01 | 2.78 | 2.18 |