Code Generation on HumanEval (SR, tau)
6.06Success Rate (SR)GTO
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| GTOModel=Vicuna-1.3 13B, Temperature=02025.09 | 6.06 | 7.95 | — | |
| EAGLE-3Model=Vicuna-1.3 13B, Temperature=02025.09 | 5.61 | 7.33 | — | |
| EAGLE-3Model=Vicuna-1.3 13B, Temperature=12025.09 | 4.77 | 6.52 | — | |
| GTOModel=Vicuna-1.3 13B, Temperature=12025.09 | 4.77 | 6.52 | — | |
| EAGLE2Model=Vicuna-1.3 13B, Temperature=02025.09 | 4.71 | 5.37 | — | |
| GTOModel=LLaMA-3.3 Instruct 70B, Temperature=02025.09 | 4.68 | 6.51 | — | |
| EAGLE2Model=Vicuna-1.3 13B, Temperature=12025.09 | 4.61 | 6.36 | — | |
| EAGLE-3Model=LLaMA-3.3 Instruct 70B, Temperature=02025.09 | 4.41 | 6.26 | — | |
| GTOModel=LLaMA-3.3 Instruct 70B, Temperature=12025.09 | 4.21 | 6.2 | — | |
| GTOModel=LLaMA-3.1 Instruct 8B, Temperature=02025.09 | 4.17 | 6.95 | — | |
| EAGLE-3Model=LLaMA-3.3 Instruct 70B, Temperature=12025.09 | 4.05 | 5.85 | — | |
| GTOModel=DeepSeek-R1 Distill-LLaMA 8B, Temperature=02025.09 | 3.98 | 6.58 | — | |
| EAGLEModel=Vicuna-1.3 13B, Temperature=12025.09 | 3.84 | 4.87 | — | |
| GRIFFINModel=LLaMA-3.1 Instruct 8B, Temperature=02025.09 | 3.73 | 5.9 | — | |
| EAGLE-3Model=LLaMA-3.1 Instruct 8B, Temperature=02025.09 | 3.68 | 6.41 | — | |
| EAGLE-3Model=DeepSeek-R1 Distill-LLaMA 8B, Temperature=02025.09 | 3.59 | 5.88 | — | |
| GTOModel=DeepSeek-R1 Distill-LLaMA 8B, Temperature=12025.09 | 3.34 | 5.44 | — | |
| EAGLE-2Model=LLaMA-3.1 Instruct 8B, Temperature=02025.09 | 3.31 | 4.7 | — | |
| GRIFFINModel=LLaMA-3.1 Instruct 8B, Temperature=12025.09 | 3.24 | 5.39 | — | |
| EAGLEModel=Vicuna-1.3 13B, Temperature=02025.09 | 3.23 | 4.12 | — | |
| GRIFFINModel=DeepSeek-R1 Distill-LLaMA 8B, Temperature=02025.09 | 3.19 | 5.23 | — | |
| GTOModel=LLaMA-3.1 Instruct 8B, Temperature=12025.09 | 3.17 | 5.92 | — | |
| EAGLE-3Model=DeepSeek-R1 Distill-LLaMA 8B, Temperature=12025.09 | 3.15 | 5.1 | — | |
| EAGLE-3Model=LLaMA-3.1 Instruct 8B, Temperature=12025.09 | 3.07 | 5.73 | — | |
| HydraModel=Vicuna-1.3 13B, Temperature=02025.09 | 3.02 | 3.68 | — | |
| GRIFFINModel=DeepSeek-R1 Distill-LLaMA 8B, Temperature=12025.09 | 2.83 | 4.68 | — | |
| GTOModel=Qwen-3 8B, Temperature=02025.09 | 2.78 | 5.37 | — | |
| EAGLE-2Model=LLaMA-3.1 Instruct 8B, Temperature=12025.09 | 2.62 | 4.37 | — | |
| EAGLE-3Model=Qwen-3 8B, Temperature=02025.09 | 2.56 | 4.88 | — | |
| GTOModel=Qwen-3 8B, Temperature=12025.09 | 2.49 | 5.03 | — | |
| HydraModel=Vicuna-1.3 13B, Temperature=12025.09 | 2.48 | 3.46 | — | |
| EAGLEModel=LLaMA-3.1 Instruct 8B, Temperature=02025.09 | 2.43 | 3.26 | — | |
| MedusaModel=Vicuna-1.3 13B, Temperature=02025.09 | 2.37 | 2.74 | — | |
| EAGLE-3Model=Qwen-3 8B, Temperature=12025.09 | 2.32 | 4.77 | — | |
| SPSModel=Vicuna-1.3 13B, Temperature=02025.09 | 2.18 | 2.52 | — | |
| EAGLEModel=LLaMA-3.1 Instruct 8B, Temperature=12025.09 | 1.97 | 3.08 | — | |
| SPSModel=Vicuna-1.3 13B, Temperature=12025.09 | 1.73 | 1.99 | — | |
| LookaheadModel=LLaMA-3.1 Instruct 8B, Temperature=02025.09 | 1.72 | 1.78 | — | |
| LookaheadModel=LLaMA-3.3 Instruct 70B, Temperature=02025.09 | 1.71 | 1.79 | — | |
| PLDModel=LLaMA-3.1 Instruct 8B, Temperature=02025.09 | 1.69 | 1.73 | — | |
| LookaheadModel=DeepSeek-R1 Distill-LLaMA 8B, Temperature=02025.09 | 1.64 | 1.71 | — | |
| PLDModel=LLaMA-3.3 Instruct 70B, Temperature=02025.09 | 1.58 | 1.67 | — | |
| PLDModel=DeepSeek-R1 Distill-LLaMA 8B, Temperature=02025.09 | 1.53 | 1.62 | — | |
| Llama 3.2-3Bγ (loss-decay constant)=None, αf (first-error focal coefficient)=0, αc (chain-loss coefficient)=02026.06 | — | — | 3.778 | |
| Llama 3.2-3Bγ (loss-decay constant)=10, αf (first-error focal coefficient)=0, αc (chain-loss coefficient)=02026.06 | — | — | 4.096 | |
| Llama 3.2-3Bγ (loss-decay constant)=10, αf (first-error focal coefficient)=0.3, αc (chain-loss coefficient)=02026.06 | — | — | 4.347 | |
| Llama 3.2-3Bγ (loss-decay constant)=10, αf (first-error focal coefficient)=0.3, αc (chain-loss coefficient)=402026.06 | — | — | 5.196 | |
| Llama-3-8Bγ (loss-decay constant)=None, αf (first-error focal coefficient)=0, αc (chain-loss coefficient)=02026.06 | — | — | 2.803 | |
| Llama-3-8Bγ (loss-decay constant)=10, αf (first-error focal coefficient)=0, αc (chain-loss coefficient)=02026.06 | — | — | 3.126 | |
| Llama-3-8Bγ (loss-decay constant)=10, αf (first-error focal coefficient)=0.3, αc (chain-loss coefficient)=02026.06 | — | — | 3.521 | |
| Llama-3-8Bγ (loss-decay constant)=10, αf (first-error focal coefficient)=0.3, αc (chain-loss coefficient)=402026.06 | — | — | 4.336 | |
| Qwen3-4Bγ (loss-decay constant)=None, αf (first-error focal coefficient)=0, αc (chain-loss coefficient)=02026.06 | — | — | 2.738 | |
| Qwen3-4Bγ (loss-decay constant)=10, αf (first-error focal coefficient)=0, αc (chain-loss coefficient)=02026.06 | — | — | 3.002 | |
| Qwen3-4Bγ (loss-decay constant)=10, αf (first-error focal coefficient)=0.3, αc (chain-loss coefficient)=02026.06 | — | — | 3.126 | |
| Qwen3-4Bγ (loss-decay constant)=10, αf (first-error focal coefficient)=0.3, αc (chain-loss coefficient)=402026.06 | — | — | 3.507 | |
| Qwen3-8Bγ (loss-decay constant)=None, αf (first-error focal coefficient)=0, αc (chain-loss coefficient)=02026.06 | — | — | 2.48 | |
| Qwen3-8Bγ (loss-decay constant)=10, αf (first-error focal coefficient)=0, αc (chain-loss coefficient)=02026.06 | — | — | 2.779 | |
| Qwen3-8Bγ (loss-decay constant)=10, αf (first-error focal coefficient)=0.3, αc (chain-loss coefficient)=02026.06 | — | — | 2.969 | |
| Qwen3-8Bγ (loss-decay constant)=10, αf (first-error focal coefficient)=0.3, αc (chain-loss coefficient)=402026.06 | — | — | 3.384 |