Code Generation on HumanEval (Pass@10)
93.1Pass@10Qwen-2.5-7B-Instruct
Evaluation Results
| Method | Links | |
|---|---|---|
| Qwen-2.5-7B-InstructDecoding Method=tree-based typical decoding, Samples=20, Temperature=0.82025.11 | 93.1 | |
| Ministral-8B-InstructDecoding Method=tree-based typical decoding, Samples=20, Temperature=0.82025.11 | 91 | |
| Llama-3.1-8B-InstructDecoding Method=tree-based typical decoding, Samples=20, Temperature=0.82025.11 | 86.3 | |
| Tulu 3 8BDecoding Method=tree-based typical decoding, Samples=20, Temperature=0.82025.11 | 83.9 | |
| EvaByte-SFTDecoding Method=tree-based typical decoding, Samples=20, Temperature=0.82025.11 | 73.7 | |
| MAP-Neo-7B-InstructDecoding Method=tree-based typical decoding, Samples=20, Temperature=0.82025.11 | 72.1 | |
| Gemma-2-9B-itDecoding Method=tree-based typical decoding, Samples=20, Temperature=0.82025.11 | 71.7 | |
| OLMo-2-7B-1124-InstructDecoding Method=tree-based typical decoding, Samples=20, Temperature=0.82025.11 | 67.6 | |
| OLMo-2-7B-SFTDecoding Method=tree-based typical decoding, Samples=20, Temperature=0.82025.11 | 67 | |
| OLMoE-1B-7B-0924-InstructDecoding Method=tree-based typical decoding, Samples=20, Temperature=0.82025.11 | 54.8 | |
| OLMo-v1.7-7B-InstructDecoding Method=tree-based typical decoding, Samples=20, Temperature=0.82025.11 | 49.7 | |
| OLMo-7B-InstructDecoding Method=tree-based typical decoding, Samples=20, Temperature=0.82025.11 | 28.7 |