Code Generation on ClassEval (Spd, tau)
4.28SpdGOOSE
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| GOOSEModel=Qwen2-8B, GPU=NVIDIA A402026.04 | 4.28 | 5.19 | |
| GOOSEModel=Llama-3-8B, GPU=NVIDIA A402026.04 | 3.82 | 4.89 | |
| TRModel=Qwen2-8B, GPU=NVIDIA A402026.04 | 3.77 | 4.5 | |
| GOOSEModel=Vicuna-13B, GPU=NVIDIA A402026.04 | 3.68 | 4.59 | |
| EAGLE-2Model=Vicuna-13B, GPU=NVIDIA A402026.04 | 3.62 | 6.03 | |
| GOOSEModel=Vicuna-7B, GPU=NVIDIA A402026.04 | 3.58 | 4.64 | |
| EAGLE-2Model=Vicuna-7B, GPU=NVIDIA A402026.04 | 3.14 | 5.5 | |
| TRModel=Llama-3-8B, GPU=NVIDIA A402026.04 | 2.87 | 3.64 | |
| GOOSEModel=Vicuna-33B, GPU=NVIDIA A1002026.04 | 2.8 | 3.69 | |
| TRModel=Vicuna-13B, GPU=NVIDIA A402026.04 | 2.71 | 3.34 | |
| TRModel=Vicuna-7B, GPU=NVIDIA A402026.04 | 2.58 | 3.29 | |
| EAGLE-2Model=Llama-3-8B, GPU=NVIDIA A402026.04 | 2.58 | 5.29 | |
| PLDModel=Qwen2-8B, GPU=NVIDIA A402026.04 | 2.5 | 2.61 | |
| LookaheadModel=Qwen2-8B, GPU=NVIDIA A402026.04 | 2.42 | 2.42 | |
| LookaheadModel=Llama-3-8B, GPU=NVIDIA A402026.04 | 2.35 | 2.57 | |
| PLDModel=Vicuna-7B, GPU=NVIDIA A402026.04 | 2.34 | 2.5 | |
| PLDModel=Vicuna-13B, GPU=NVIDIA A402026.04 | 2.34 | 2.47 | |
| PLDModel=Llama-3-8B, GPU=NVIDIA A402026.04 | 2.3 | 2.49 | |
| LookaheadModel=Vicuna-7B, GPU=NVIDIA A402026.04 | 2.27 | 2.41 | |
| LookaheadModel=Vicuna-13B, GPU=NVIDIA A402026.04 | 2.14 | 2.26 | |
| TRModel=Vicuna-33B, GPU=NVIDIA A1002026.04 | 1.97 | 2.62 | |
| LookaheadModel=Vicuna-33B, GPU=NVIDIA A1002026.04 | 1.86 | 2.02 | |
| PLDModel=Vicuna-33B, GPU=NVIDIA A1002026.04 | 1.85 | 1.99 | |
| RESTModel=Vicuna-7B, GPU=NVIDIA A402026.04 | 1.68 | 1.73 | |
| RESTModel=Vicuna-13B, GPU=NVIDIA A402026.04 | 1.61 | 1.66 | |
| RESTModel=Vicuna-33B, GPU=NVIDIA A1002026.04 | 1.48 | 1.54 | |
| RESTModel=Llama-3-8B, GPU=NVIDIA A402026.04 | 1.33 | 1.35 | |
| RESTModel=Qwen2-8B, GPU=NVIDIA A402026.04 | 1.22 | 1.14 | |
| ARModel=Vicuna-7B, GPU=NVIDIA A402026.04 | 1 | 1 | |
| ARModel=Llama-3-8B, GPU=NVIDIA A402026.04 | 1 | 1 | |
| ARModel=Qwen2-8B, GPU=NVIDIA A402026.04 | 1 | 1 | |
| ARModel=Vicuna-13B, GPU=NVIDIA A402026.04 | 1 | 1 | |
| ARModel=Vicuna-33B, GPU=NVIDIA A1002026.04 | 1 | 1 |