Software Engineering on SWE-Bench Lite (Acc, Cost)
57.09AccuracyCoT-SC
Evaluation results
| Method | Links | ||
|---|---|---|---|
| CoT-SCLLM Backbone=GPT-52026.06 | 57.09 | 286.4 | |
| DyLANLLM Backbone=GPT-52026.06 | 55.97 | 227.4 | |
| MAS-ZeroLLM Backbone=GPT-52026.06 | 45.52 | 998.2 | |
| CoTLLM Backbone=GPT-52026.06 | 43.91 | 14.9 | |
| MAS-OrchestraLLM Backbone=GPT-52026.06 | 41.79 | 83.5 | |
| AFlowLLM Backbone=GPT-52026.06 | 39.05 | 997.5 | |
| MaASLLM Backbone=GPT-52026.06 | 32.71 | 95.4 | |
| CoTLLM Backbone=Gemini-2.5-Pro2026.06 | 28.7 | 47.17 | |
| DyLANLLM Backbone=Gemini-2.5-Pro2026.06 | 27.6 | 327.4 | |
| ADASLLM Backbone=GPT-52026.06 | 27.23 | 124.5 | |
| CoT-SCLLM Backbone=Gemini-2.5-Pro2026.06 | 26.5 | 240.56 | |
| ADASLLM Backbone=Gemini-2.5-Pro2026.06 | 26.5 | 214 | |
| CoT-SCLLM Backbone=GPT-4o2026.06 | 22.01 | 168.3 | |
| MaASLLM Backbone=Gemini-2.5-Pro2026.06 | 20.5 | 206.27 | |
| CoTLLM Backbone=GPT-4o2026.06 | 20.02 | 25.6 | |
| DyLANLLM Backbone=GPT-4o2026.06 | 19.28 | 230.8 | |
| MAS-OrchestraLLM Backbone=GPT-4o2026.06 | 17.55 | 82.7 | |
| MAS-ZeroLLM Backbone=Gemini-2.5-Pro2026.06 | 17.5 | — | |
| MAS-ZeroLLM Backbone=GPT-4o2026.06 | 15.17 | 1,210.9 | |
| MaASLLM Backbone=GPT-4o2026.06 | 12.19 | 35.6 | |
| AFlowLLM Backbone=GPT-4o2026.06 | 10.57 | 903.2 | |
| ADASLLM Backbone=GPT-4o2026.06 | 8.97 | 104.4 | |
| AFlowLLM Backbone=Gemini-2.5-Pro2026.06 | 0 | 434 |