Code Completion on HumanEval
66.5HE ScoreOpenCoder-8B-Base
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| OpenCoder-8B-BaseSize=8B2024.11 | 66.5 | 63.4 | — | — | |
| Qwen2.5-Coder-7BSize=7B2024.11 | 61.6 | 53 | — | — | |
| OpenCoder-1.5B-BaseSize=1.5B2024.11 | 54.3 | 49.4 | — | — | |
| Yi-Coder-9BSize=9B2024.11 | 53.7 | 46.3 | — | — | |
| CodeQwen1.5-7BSize=7B2024.11 | 51.8 | 45.7 | — | — | |
| DS-Coder-6.7B-BaseSize=6.7B2024.11 | 47.6 | 39.6 | — | — | |
| StarCoder2-15BSize=15B2024.11 | 46.3 | 37.8 | — | — | |
| Qwen2.5-Coder-1.5BSize=1.5B2024.11 | 43.9 | 36.6 | — | — | |
| Yi-Coder-1.5BSize=1.5B2024.11 | 41.5 | 32.9 | — | — | |
| DS-Coder-V2-Lite-Base (MoE)Size=16B2024.11 | 40.9 | 34.1 | — | — | |
| CodeGemma-7BSize=7B2024.11 | 39 | 32.3 | — | — | |
| StarCoder2-7BSize=7B2024.11 | 35.4 | 29.9 | — | — | |
| DeepSeek-Coder-1.3B-BaseSize=1.3B2024.11 | 34.8 | 26.8 | — | — | |
| CodeLlama-7BSize=7B2024.11 | 33.5 | 26.2 | — | — | |
| StarCoder2-3BSize=3B2024.11 | 31.7 | 27.4 | — | — | |
| CodeGemma-2BSize=2B2024.11 | 31.1 | 16.5 | — | — | |
| Crystal-7BSize=7B2024.11 | 22.6 | 20.7 | — | — | |
| EAGLE-2Model=LLaMA-2, Temp=02026.02 | — | — | 4.78 | 283.84 | |
| HASSModel=LLaMA-2, Temp=02026.02 | — | — | 5.15 | 305.31 | |
| PRISMModel=LLaMA-2, Temp=02026.02 | — | — | 5.33 | 316.16 | |
| PRISMModel=LLaMA-3, Temp=02026.02 | — | — | 5.43 | 251.96 | |
| PRISMModel=LLaMA-3, Temp=12026.02 | — | — | 5.02 | 216.3 | |
| Standard Speculative DecodingModel=LLaMA-2, Temp=02026.02 | — | — | 2.7 | 145.65 | |
| VanillaModel=LLaMA-2, Temp=02026.02 | — | — | — | 96.1 |