Code Generation on DataGovbench Operator-Level 1.0 (test)
41.47ATSo4-mini
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| o4-miniAvg. Tokens=1510.68, Generation Time (s)=1127.16, Execution Time (s)=167.282025.12 | 41.47 | 49 | 68 | 52.82 | |
| GPT-5Avg. Tokens=3706.21, Generation Time (s)=3069.44, Execution Time (s)=598.732025.12 | 40.98 | 49 | 81 | 56.99 | |
| Gemini-2.5-flashAvg. Tokens=5234.3, Generation Time (s)=5727.56, Execution Time (s)=355.652025.12 | 40.26 | 48 | 80 | 56.09 | |
| Kimi-K2-instructAvg. Tokens=721.16, Generation Time (s)=864.21, Execution Time (s)=652.622025.12 | 39.52 | 49 | 70 | 52.84 | |
| Claude-4-opusAvg. Tokens=1390.04, Generation Time (s)=3298.22, Execution Time (s)=158.852025.12 | 38.3 | 47 | 79 | 54.77 | |
| Grok-4Avg. Tokens=4575.07, Generation Time (s)=7700.3, Execution Time (s)=406.622025.12 | 36.9 | 44 | 67 | 49.3 | |
| Claude-4-sonnetAvg. Tokens=1672.91, Generation Time (s)=3149.83, Execution Time (s)=229.72025.12 | 36.75 | 46 | 85 | 55.92 | |
| Grok-3Avg. Tokens=688.51, Generation Time (s)=811.22, Execution Time (s)=685.252025.12 | 35.41 | 44 | 71 | 50.14 | |
| o3Avg. Tokens=1415.08, Generation Time (s)=1291.82, Execution Time (s)=35.162025.12 | 34.48 | 45 | 63 | 47.49 | |
| o1Avg. Tokens=1908.54, Generation Time (s)=3916.7, Execution Time (s)=35.552025.12 | 32.5 | 41 | 74 | 49.17 | |
| GPT-4oAvg. Tokens=555.26, Generation Time (s)=431.85, Execution Time (s)=29.722025.12 | 32.04 | 41 | 56 | 43.01 |