Code Generation on CoderEval-Python file-runnable
30.29Pass@1COCOGEN
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| COCOGENBase LLM=GPT-3.5-Turbo2024.03 | 30.29 | 43.58 | 47.06 | |
| RepoCoderBase LLM=GPT-3.5-Turbo2024.03 | 29.41 | 34.61 | 36.76 | |
| DirectBase LLM=GPT-3.5-Turbo2024.03 | 19.85 | 27.62 | 30.88 | |
| DirectBase LLM=Code Llama (13B)2024.03 | 18.53 | 27.82 | 29.41 | |
| ReACCBase LLM=GPT-3.5-Turbo2024.03 | 17.65 | 28.92 | 33.82 | |
| ReACCBase LLM=Code Llama (13B)2024.03 | 17.65 | 27.61 | 33.82 | |
| COCOGENBase LLM=Code Llama (13B)2024.03 | 17.06 | 29.39 | 33.82 | |
| RepoCoderBase LLM=Code Llama (13B)2024.03 | 15 | 28.31 | 32.35 |