Code Generation on SimuPhy
99.9Avg Executable Rate @8 (E.R.)DeepSeek-V3.1
Evaluation results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| DeepSeek-V3.1Size=671B2026.02 | 99.9 | 95.2 | 64 | 14.5 | 100 | 99.4 | 90.4 | 40.7 | |
| GPT-oss-120bSize=120B2026.02 | 99.8 | 93.7 | 76 | 14 | 100 | 99.7 | 91.9 | 37.4 | |
| Qwen3-32BSize=32B2026.02 | 99.8 | 96.6 | 51 | 11.1 | 100 | 100 | 82.6 | 30.5 | |
| Qwen3-235B-A22BSize=235B2026.02 | 99.7 | 95.9 | 68.5 | 15.1 | 100 | 100 | 91.3 | 41 | |
| DeepSeek-R1-0528Size=671B2026.02 | 99.7 | 97.3 | 77 | 21.5 | 100 | 100 | 93.4 | 52.7 | |
| Gemini-2.5-proSize=-2026.02 | 99.1 | 77.3 | 38 | 12.7 | 100 | 99.7 | 77.8 | 37.4 | |
| GPT-oss-20bSize=21.5B2026.02 | 99.1 | 92 | 61.6 | 10.5 | 100 | 100 | 85.6 | 32 | |
| GPT-o3Size=-2026.02 | 98.9 | 75.2 | 64.1 | 15.9 | 100 | 100 | 95.5 | 45.2 | |
| GPT-5-mediumSize=-2026.02 | 97.2 | 68.9 | 55.7 | 20.5 | 100 | 100 | 93.1 | 59.9 | |
| GPT-o4-miniSize=-2026.02 | 88.1 | 86.9 | 78.6 | 17.2 | 100 | 100 | 95.5 | 42.5 |