Python program generation for compositional functions on Base Experiment Random setting
318.13L(P+)o3-mini
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| o3-miniversion=o3-mini-2025-01-31, type=reasoning model2026.04 | 318.13 | 0.07 | 0.67 | |
| Gemini-2.0version=Gemini-2.0-flash, type=non-reasoning model2026.04 | 295 | 1.17 | 3.71 | |
| Gemini-2.5version=Gemini-2.5-pro-exp-03-25, type=reasoning model2026.04 | 292 | 0.1 | 10 | |
| o1-miniversion=o1-mini-2024-09-12, type=reasoning model2026.04 | 278.53 | 3.2 | 0 | |
| DeepSeek-R1version=DeepSeek-R1-0120, type=reasoning model2026.04 | 270 | 5.13 | 0 | |
| DeepSeek-V3version=DeepSeek-V3-0324, type=non-reasoning model2026.04 | 239.7 | 5.9 | 3.33 | |
| Claude-3.7version=Claude-3.7-Sonnet-20250219, type=reasoning model2026.04 | 192.33 | 8.63 | 3.52 | |
| QwQ-Plusversion=QwQ-Plus-0305, type=reasoning model2026.04 | 181.27 | 9.2 | 0 | |
| Claude-3.5version=Claude-3.5-Haiku-20241022, type=non-reasoning model2026.04 | 133.97 | 15.43 | 0 | |
| GPT-4oversion=GPT-4o-2024-08-06, type=non-reasoning model2026.04 | 111.33 | 15.93 | 0 | |
| Qwen-Maxversion=Qwen-Max-0125, type=non-reasoning model2026.04 | 62 | 14.77 | 0 |