Code Generation on BIRD-Python Original (dev)
0.6584Execution Accuracy (Simple)Qwen3 (14B)
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| Qwen3 (14B)Reasoning=true2026.01 | 0.6584 | 0.3233 | 0.2966 | 0.4289 | |
| DeepSeek-R1Reasoning=true2026.01 | 0.6238 | 0.4504 | 0.4828 | 0.5574 | |
| Qwen3-MaxReasoning=false2026.01 | 0.6227 | 0.4375 | 0.4759 | 0.5528 | |
| Qwen3-Coder-PlusReasoning=false2026.01 | 0.6032 | 0.4526 | 0.4207 | 0.5398 | |
| Qwen3 (32B)Reasoning=true2026.01 | 0.6022 | 0.4224 | 0.4552 | 0.5332 | |
| Qwen-MaxReasoning=false2026.01 | 0.5957 | 0.4504 | 0.4276 | 0.5359 | |
| Qwen2.5-Coder (32B)Reasoning=false2026.01 | 0.5838 | 0.4181 | 0.3793 | 0.5143 | |
| Qwen3 (7B)Reasoning=true2026.01 | 0.547 | 0.3621 | 0.3448 | 0.472 | |
| Qwen2.5-Coder (14B)Reasoning=false2026.01 | 0.5124 | 0.3384 | 0.3586 | 0.4452 | |
| Qwen3-Coder (30B)Reasoning=false2026.01 | 0.4692 | 0.3405 | 0.3241 | 0.4153 | |
| Qwen2.5-Coder (7B)Reasoning=false2026.01 | 0.3989 | 0.1897 | 0.2345 | 0.3194 | |
| TableGPT2 (7B)Reasoning=false2026.01 | 0.2315 | 0.1509 | 0.1103 | 0.1745 | |
| AutoCoder (7B)Reasoning=false2026.01 | 0.075 | 0.0388 | 0.0172 | 0.0488 | |
| StarCoder2 (7B)Reasoning=false2026.01 | 0.0162 | 0.0043 | 0 | 0.0085 |