Code Generation on BIRD-Python Verified
0.6995Execution Accuracy (Simple)DeepSeek-R1
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| DeepSeek-R1Reasoning=true2026.01 | 0.6995 | 0.5302 | 0.4552 | 0.6252 | |
| Qwen3-MaxReasoning=false2026.01 | 0.6789 | 0.5991 | 0.4621 | 0.6343 | |
| Qwen3-Coder-PlusReasoning=false2026.01 | 0.6595 | 0.5582 | 0.531 | 0.616 | |
| Qwen-MaxReasoning=false2026.01 | 0.6573 | 0.5366 | 0.5241 | 0.6082 | |
| Qwen3 (32B)Reasoning=true2026.01 | 0.6533 | 0.5194 | 0.5448 | 0.601 | |
| Qwen2.5-Coder (32B)Reasoning=false2026.01 | 0.6216 | 0.4978 | 0.4276 | 0.5658 | |
| Qwen2.5-Coder (14B)Reasoning=false2026.01 | 0.6076 | 0.4569 | 0.4069 | 0.543 | |
| Qwen3 (7B)Reasoning=true2026.01 | 0.5946 | 0.4504 | 0.3931 | 0.5319 | |
| Qwen3 (14B)Reasoning=true2026.01 | 0.547 | 0.3793 | 0.3724 | 0.4798 | |
| Qwen3-Coder (30B)Reasoning=false2026.01 | 0.4951 | 0.4159 | 0.3586 | 0.4583 | |
| Qwen2.5-Coder (7B)Reasoning=false2026.01 | 0.4191 | 0.2651 | 0.2828 | 0.3595 | |
| TableGPT2 (7B)Reasoning=false2026.01 | 0.2546 | 0.1724 | 0.131 | 0.1982 | |
| AutoCoder (7B)Reasoning=false2026.01 | 0.0865 | 0.0431 | 0.0207 | 0.0545 | |
| StarCoder2 (7B)Reasoning=false2026.01 | 0.0195 | 0.0086 | 0 | 0.0112 |