Code Generation on CoderEval-Python class-runnable
35.45Pass@1RepoCoder
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| RepoCoderBase LLM=GPT-3.5-Turbo2024.03 | 35.45 | 40.46 | 41.82 | |
| COCOGENBase LLM=GPT-3.5-Turbo2024.03 | 28 | 44.92 | 49.09 | |
| COCOGENBase LLM=Code Llama (13B)2024.03 | 26.36 | 39.42 | 41.82 | |
| COCOGENBase LLM=GPT-3.5-Turbo2024.03 | 21.3 | 36.73 | 39.13 | |
| ReACCBase LLM=GPT-3.5-Turbo2024.03 | 20.36 | 33.27 | 38.18 | |
| ReACCBase LLM=Code Llama (13B)2024.03 | 20.36 | 33.27 | 38.18 | |
| DirectBase LLM=Code Llama (13B)2024.03 | 18.91 | 30.65 | 34.55 | |
| RepoCoderBase LLM=Code Llama (13B)2024.03 | 17.82 | 35.22 | 40 | |
| RepoCoderBase LLM=GPT-3.5-Turbo2024.03 | 16.96 | 19.57 | 21.74 | |
| RepoCoderBase LLM=Code Llama (13B)2024.03 | 16.09 | 21.36 | 21.74 | |
| COCOGENBase LLM=Code Llama (13B)2024.03 | 13.04 | 28.04 | 34.78 | |
| ReACCBase LLM=GPT-3.5-Turbo2024.03 | 11.3 | 19.53 | 21.74 | |
| ReACCBase LLM=Code Llama (13B)2024.03 | 11.3 | 19.53 | 21.74 | |
| DirectBase LLM=GPT-3.5-Turbo2024.03 | 9.57 | 12.08 | 13.04 | |
| DirectBase LLM=GPT-3.5-Turbo2024.03 | 8.73 | 12.57 | 14.55 | |
| DirectBase LLM=Code Llama (13B)2024.03 | 5.22 | 8.7 | 13.04 |