Code Generation on HumanEval (Acc, Tp, Spd)
89AccuracySpecDecode
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| SpecDecodeModel Series=Qwen-2.5 (72B/7B), Shot=0-shot, Protocol=Pass@12026.04 | 89 | 26.9 | 1.81 | |
| Vanilla DecodingModel Series=Qwen-2.5 (72B/7B), Shot=0-shot, Protocol=Pass@12026.04 | 88.4 | 14.9 | 1 | |
| CSDModel Series=Qwen-2.5 (72B/7B), Shot=0-shot, Protocol=Pass@12026.04 | 88.4 | 29 | 1.95 | |
| CSDModel Series=Llama-3 (70B/1B), Shot=0-shot, Protocol=Pass@12026.04 | 79.3 | 37 | 2.33 | |
| Lossy SDModel Series=Llama-3 (70B/1B), Shot=0-shot, Protocol=Pass@1, τ=0.62026.04 | 78 | 32.3 | 2.04 | |
| Vanilla DecodingModel Series=Llama-3 (70B/1B), Shot=0-shot, Protocol=Pass@12026.04 | 76.8 | 15.9 | 1 | |
| SpecDecodeModel Series=Llama-3 (70B/1B), Shot=0-shot, Protocol=Pass@12026.04 | 76.8 | 30.1 | 1.9 | |
| SWIFTModel Series=Llama-3 (70B/1B), Shot=0-shot, Protocol=Pass@12026.04 | 76.8 | 17.5 | 1.1 | |
| LookaheadModel Series=Llama-3 (70B/1B), Shot=0-shot, Protocol=Pass@12026.04 | 76.8 | 14.4 | 0.91 |