Repo-level code generation on CODEAGENTBENCH
37.6Pass@1GPT-4-turbo
Evaluation results
| Method | Links | |
|---|---|---|
| GPT-4-turboStrategy=Rule-based, LLM Type=Closed source LLM2024.01 | 37.6 | |
| GPT-4-turboStrategy=ReAct, LLM Type=Closed source LLM2024.01 | 34.7 | |
| GPT-4-turboStrategy=OpenAIFunc, LLM Type=Closed source LLM2024.01 | 34.7 | |
| GPT-3.5-turboStrategy=Rule-based, LLM Type=Closed source LLM2024.01 | 31.7 | |
| GPT-3.5-turboStrategy=ReAct, LLM Type=Closed source LLM2024.01 | 30.7 | |
| GPT-3.5-turboStrategy=OpenAIFunc, LLM Type=Closed source LLM2024.01 | 28.7 | |
| GPT-4-turboStrategy=Tool-Planning, LLM Type=Closed source LLM2024.01 | 25.7 | |
| GPT-3-davinciStrategy=Rule-based, Scales=175B, LLM Type=Closed source LLM2024.01 | 24.8 | |
| DeepSeek-33BStrategy=Rule-based, Scales=33B, LLM Type=Open source LLM2024.01 | 24.8 | |
| GPT-3-davinciStrategy=ReAct, Scales=175B, LLM Type=Closed source LLM2024.01 | 22.8 | |
| GPT-3.5-turboStrategy=Tool-Planning, LLM Type=Closed source LLM2024.01 | 21.8 | |
| GPT-4-turboStrategy=NoAgent, LLM Type=Closed source LLM2024.01 | 21.8 | |
| DeepSeek-33BStrategy=ReAct, Scales=33B, LLM Type=Open source LLM2024.01 | 20.8 | |
| GPT-3.5-turboStrategy=NoAgent, LLM Type=Closed source LLM2024.01 | 19.8 | |
| GPT-3-davinciStrategy=Tool-Planning, Scales=175B, LLM Type=Closed source LLM2024.01 | 18.8 | |
| GPT-3-davinciStrategy=NoAgent, Scales=175B, LLM Type=Closed source LLM2024.01 | 16.8 | |
| DeepSeek-33BStrategy=Tool-Planning, Scales=33B, LLM Type=Open source LLM2024.01 | 15.8 | |
| DeepSeek-33BStrategy=NoAgent, Scales=33B, LLM Type=Open source LLM2024.01 | 13.9 | |
| Llama2-70B-chatStrategy=Rule-based, Scales=70B, LLM Type=Open source LLM2024.01 | 12.9 | |
| Llama2-70B-chatStrategy=ReAct, Scales=70B, LLM Type=Open source LLM2024.01 | 11.9 | |
| Llama2-70B-chatStrategy=Tool-Planning, Scales=70B, LLM Type=Open source LLM2024.01 | 11.9 | |
| Claude-2Strategy=Rule-based, LLM Type=Closed source LLM2024.01 | 10.9 | |
| Llama2-70B-chatStrategy=NoAgent, Scales=70B, LLM Type=Open source LLM2024.01 | 10.9 | |
| Claude-2Strategy=ReAct, LLM Type=Closed source LLM2024.01 | 9.9 | |
| Claude-2Strategy=Tool-Planning, LLM Type=Closed source LLM2024.01 | 9.9 | |
| Claude-2Strategy=NoAgent, LLM Type=Closed source LLM2024.01 | 8.9 | |
| WizardCoder-34BStrategy=Rule-based, Scales=34B, LLM Type=Open source LLM2024.01 | 6.9 | |
| Code Llama-34BStrategy=Rule-based, Scales=34B, LLM Type=Open source LLM2024.01 | 5 | |
| WizardCoder-34BStrategy=ReAct, Scales=34B, LLM Type=Open source LLM2024.01 | 5 | |
| Code Llama-34BStrategy=ReAct, Scales=34B, LLM Type=Open source LLM2024.01 | 4 | |
| Code Llama-34BStrategy=Tool-Planning, Scales=34B, LLM Type=Open source LLM2024.01 | 4 | |
| WizardCoder-34BStrategy=Tool-Planning, Scales=34B, LLM Type=Open source LLM2024.01 | 4 | |
| Code Llama-34BStrategy=NoAgent, Scales=34B, LLM Type=Open source LLM2024.01 | 2 | |
| WizardCoder-34BStrategy=NoAgent, Scales=34B, LLM Type=Open source LLM2024.01 | 2 | |
| Vicuna-13BStrategy=NoAgent, Scales=13B, LLM Type=Open source LLM2024.01 | 1 | |
| Vicuna-13BStrategy=Rule-based, Scales=13B, LLM Type=Open source LLM2024.01 | 1 | |
| Vicuna-13BStrategy=ReAct, Scales=13B, LLM Type=Open source LLM2024.01 | 0 | |
| Vicuna-13BStrategy=Tool-Planning, Scales=13B, LLM Type=Open source LLM2024.01 | 0 |