Repo-level code generation on CODEAGENTBENCH benchmark leaderboard · SOTA2 Research

Repo-level code generation on CODEAGENTBENCH

37.6Pass@1GPT-4-turbo

Evaluation Results

Method	Links
GPT-4-turbo2024.01		37.6
GPT-4-turbo2024.01		34.7
GPT-4-turbo2024.01		34.7
GPT-3.5-turbo2024.01		31.7
GPT-3.5-turbo2024.01		30.7
GPT-3.5-turbo2024.01		28.7
GPT-4-turbo2024.01		25.7
GPT-3-davinci2024.01		24.8
DeepSeek-33B2024.01		24.8
GPT-3-davinci2024.01		22.8
GPT-3.5-turbo2024.01		21.8
GPT-4-turbo2024.01		21.8
DeepSeek-33B2024.01		20.8
GPT-3.5-turbo2024.01		19.8
GPT-3-davinci2024.01		18.8
GPT-3-davinci2024.01		16.8
DeepSeek-33B2024.01		15.8
DeepSeek-33B2024.01		13.9
Llama2-70B-chat2024.01		12.9
Llama2-70B-chat2024.01		11.9
Llama2-70B-chat2024.01		11.9
Claude-22024.01		10.9
Llama2-70B-chat2024.01		10.9
Claude-22024.01		9.9
Claude-22024.01		9.9
Claude-22024.01		8.9
WizardCoder-34B2024.01		6.9
Code Llama-34B2024.01		5
WizardCoder-34B2024.01		5
Code Llama-34B2024.01		4
Code Llama-34B2024.01		4
WizardCoder-34B2024.01		4
Code Llama-34B2024.01		2
WizardCoder-34B2024.01		2
Vicuna-13B2024.01		1
Vicuna-13B2024.01		1
Vicuna-13B2024.01		0
Vicuna-13B2024.01		0