Python program generation for compositional functions on Base Experiment Horizontal setting
189.6L(P+)Gemini-2.0
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| Gemini-2.0version=Gemini-2.0-flash, type=non-reasoning model2026.04 | 189.6 | 7.17 | 7.77 | |
| DeepSeek-V3version=DeepSeek-V3-0324, type=non-reasoning model2026.04 | 165.27 | 3.2 | 42.87 | |
| Claude-3.5version=Claude-3.5-Haiku-20241022, type=non-reasoning model2026.04 | 118.43 | 14.1 | 6.69 | |
| QwQ-Plusversion=QwQ-Plus-0305, type=reasoning model2026.04 | 71.33 | 7.3 | 44 | |
| o1-miniversion=o1-mini-2024-09-12, type=reasoning model2026.04 | 49.43 | 0.3 | 94.49 | |
| o3-miniversion=o3-mini-2025-01-31, type=reasoning model2026.04 | 46.73 | 0.27 | 95.69 | |
| GPT-4oversion=GPT-4o-2024-08-06, type=non-reasoning model2026.04 | 46.67 | 15.7 | 0 | |
| Gemini-2.5version=Gemini-2.5-pro-exp-03-25, type=reasoning model2026.04 | 45.83 | 0.7 | 92.92 | |
| Qwen-Maxversion=Qwen-Max-0125, type=non-reasoning model2026.04 | 44.57 | 8.53 | 46.67 | |
| DeepSeek-R1version=DeepSeek-R1-0120, type=reasoning model2026.04 | 43.23 | 1.33 | 89.8 | |
| Claude-3.7version=Claude-3.7-Sonnet-20250219, type=reasoning model2026.04 | 40.27 | 2.2 | 84.67 |