Code Generation on BIRD-Python Verified benchmark leaderboard · SOTA2 Research

Code Generation on BIRD-Python Verified

0.6995Execution Accuracy (Simple)DeepSeek-R1

Evaluation Results

Method	Links
DeepSeek-R12026.01		0.6995	0.5302	0.4552	0.6252
Qwen3-Max2026.01		0.6789	0.5991	0.4621	0.6343
Qwen3-Coder-Plus2026.01		0.6595	0.5582	0.531	0.616
Qwen-Max2026.01		0.6573	0.5366	0.5241	0.6082
Qwen3 (32B)2026.01		0.6533	0.5194	0.5448	0.601
Qwen2.5-Coder (32B)2026.01		0.6216	0.4978	0.4276	0.5658
Qwen2.5-Coder (14B)2026.01		0.6076	0.4569	0.4069	0.543
Qwen3 (7B)2026.01		0.5946	0.4504	0.3931	0.5319
Qwen3 (14B)2026.01		0.547	0.3793	0.3724	0.4798
Qwen3-Coder (30B)2026.01		0.4951	0.4159	0.3586	0.4583
Qwen2.5-Coder (7B)2026.01		0.4191	0.2651	0.2828	0.3595
TableGPT2 (7B)2026.01		0.2546	0.1724	0.131	0.1982
AutoCoder (7B)2026.01		0.0865	0.0431	0.0207	0.0545
StarCoder2 (7B)2026.01		0.0195	0.0086	0	0.0112