Python program generation for compositional functions on Base Experiment Block setting
255.93L(P+)Gemini-2.0
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| Gemini-2.0version=Gemini-2.0-flash, type=non-reasoning model2026.04 | 255.93 | 4.5 | 0.43 | |
| DeepSeek-R1version=DeepSeek-R1-0120, type=reasoning model2026.04 | 254.43 | 3.63 | 7.62 | |
| DeepSeek-V3version=DeepSeek-V3-0324, type=non-reasoning model2026.04 | 241.53 | 6.6 | 0 | |
| o1-miniversion=o1-mini-2024-09-12, type=reasoning model2026.04 | 215.23 | 3.03 | 19.38 | |
| Gemini-2.5version=Gemini-2.5-pro-exp-03-25, type=reasoning model2026.04 | 197.03 | 0.13 | 42.96 | |
| o3-miniversion=o3-mini-2025-01-31, type=reasoning model2026.04 | 150.87 | 0.47 | 57.07 | |
| Claude-3.5version=Claude-3.5-Haiku-20241022, type=non-reasoning model2026.04 | 135.03 | 15.1 | 0 | |
| QwQ-Plusversion=QwQ-Plus-0305, type=reasoning model2026.04 | 118.33 | 11.23 | 2.86 | |
| Claude-3.7version=Claude-3.7-Sonnet-20250219, type=reasoning model2026.04 | 89.17 | 6.4 | 47.57 | |
| GPT-4oversion=GPT-4o-2024-08-06, type=non-reasoning model2026.04 | 51.8 | 15.9 | 0 | |
| Qwen-Maxversion=Qwen-Max-0125, type=non-reasoning model2026.04 | 50.7 | 15.03 | 0.48 |