Python program generation for compositional functions on Base Experiment Vertical setting
286.13L(P+)Gemini-2.0
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| Gemini-2.0version=Gemini-2.0-flash, type=non-reasoning model2026.04 | 286.13 | 2.13 | 0.67 | |
| o1-miniversion=o1-mini-2024-09-12, type=reasoning model2026.04 | 280.87 | 2.23 | 4.29 | |
| DeepSeek-V3version=DeepSeek-V3-0324, type=non-reasoning model2026.04 | 258.87 | 5.13 | 0 | |
| DeepSeek-R1version=DeepSeek-R1-0120, type=reasoning model2026.04 | 254.77 | 6 | 0 | |
| o3-miniversion=o3-mini-2025-01-31, type=reasoning model2026.04 | 243.53 | 0.07 | 27.31 | |
| Gemini-2.5version=Gemini-2.5-pro-exp-03-25, type=reasoning model2026.04 | 234.9 | 0 | 30.39 | |
| Claude-3.7version=Claude-3.7-Sonnet-20250219, type=reasoning model2026.04 | 216.3 | 5.1 | 14.71 | |
| Claude-3.5version=Claude-3.5-Haiku-20241022, type=non-reasoning model2026.04 | 120 | 16 | 0 | |
| QwQ-Plusversion=QwQ-Plus-0305, type=reasoning model2026.04 | 95.97 | 9.17 | 23.31 | |
| GPT-4oversion=GPT-4o-2024-08-06, type=non-reasoning model2026.04 | 90.63 | 15.9 | 0 | |
| Qwen-Maxversion=Qwen-Max-0125, type=non-reasoning model2026.04 | 71.8 | 14.93 | 0 |