Environment Construction on Multi-Docker-Eval SWE-Factory
17.17F2P RateGPT-OSS-20B
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| GPT-OSS-20BModel Type=Open-source, Avg input=184.23, Avg output=58.37, Avg docker=0.872026.01 | 17.17 | 29.44 | |
| Qwen3-Coder-30B-A3B-InstructModel Type=Our Method (Baselines), Avg input=150.10, Avg output=13.75, Avg docker=0.932026.01 | 19.46 | 34.13 | |
| Qwen3-235B-A22BModel Type=Open-source, Avg input=101.46, Avg output=38.87, Avg docker=1.002026.01 | 23.65 | 34.53 | |
| DeepSeek-R1Model Type=Open-source, Avg input=138.05, Avg output=60.10, Avg docker=1.022026.01 | 26.65 | 41.72 | |
| GPT-OSS-120BModel Type=Open-source, Avg input=128.31, Avg output=30.17, Avg docker=0.832026.01 | 27 | 37.72 | |
| Gemini-2.5-FlashModel Type=Closed-source, Avg input=153.43, Avg output=32.60, Avg docker=0.972026.01 | 29.44 | 40.62 | |
| GPT-5-MiniModel Type=Closed-source, Avg input=339.94, Avg output=103.32, Avg docker=0.952026.01 | 34.13 | 49.6 | |
| Claude-Sonnet-4Model Type=Closed-source, Avg input=182.85, Avg output=15.01, Avg docker=1.172026.01 | 35.53 | 47.41 | |
| Kimi-K2-thinkingModel Type=Open-source, Avg input=162.12, Avg output=59.40, Avg docker=1.052026.01 | 36.53 | 52.69 | |
| Kimi-K2-0905Model Type=Open-source, Avg input=113.02, Avg output=7.92, Avg docker=1.012026.01 | 37.62 | 55.49 | |
| DeepSeek-v3.1Model Type=Open-source, Avg input=158.11, Avg output=17.15, Avg docker=1.022026.01 | 37.72 | 52.89 | |
| DOCKSMITHModel Type=Our Method, Avg input=207.68, Avg output=26.38, Avg docker=1.132026.01 | 39.72 | 58.28 |