Software Engineering Task Completion on SWE-bench
90.2S@50 Success RateAutoGen
Evaluation results
| Method | Links | ||
|---|---|---|---|
| AutoGenN=16, Backbone=GPT-4o-mini, Topology=distinct-shard2026.05 | 90.2 | — | |
| AutoGenN=8, Backbone=GPT-4o-mini, Topology=distinct-shard2026.05 | 88.6 | — | |
| AutoGenN=4, Backbone=GPT-4o-mini, Topology=distinct-shard2026.05 | 88 | — | |
| LangGraphN=16, Backbone=GPT-4o-mini, Topology=distinct-shard2026.05 | 82.6 | — | |
| LangGraphN=8, Backbone=GPT-4o-mini, Topology=distinct-shard2026.05 | 75.9 | — | |
| S-BUSN=4, Comparison Baseline=CrewAI, Backbone=GPT-4o-mini, Topology=distinct-shard2026.05 | 73.9 | 10 | |
| S-BUSN=4, Comparison Baseline=LangGraph, Backbone=GPT-4o-mini, Topology=distinct-shard2026.05 | 73.9 | 0.244 | |
| S-BUSN=4, Comparison Baseline=AutoGen, Backbone=GPT-4o-mini, Topology=distinct-shard2026.05 | 73.9 | 0.006 | |
| S-BUSN=8, Comparison Baseline=CrewAI, Backbone=GPT-4o-mini, Topology=distinct-shard2026.05 | 71.4 | 10 | |
| S-BUSN=8, Comparison Baseline=LangGraph, Backbone=GPT-4o-mini, Topology=distinct-shard2026.05 | 71.4 | 0.203 | |
| S-BUSN=8, Comparison Baseline=AutoGen, Backbone=GPT-4o-mini, Topology=distinct-shard2026.05 | 71.4 | 0.001 | |
| S-BUSN=16, Comparison Baseline=CrewAI, Backbone=GPT-4o-mini, Topology=distinct-shard2026.05 | 70.5 | 10 | |
| S-BUSN=16, Comparison Baseline=LangGraph, Backbone=GPT-4o-mini, Topology=distinct-shard2026.05 | 70.5 | 0.012 | |
| S-BUSN=16, Comparison Baseline=AutoGen, Backbone=GPT-4o-mini, Topology=distinct-shard2026.05 | 70.5 | 10 | |
| LangGraphN=4, Backbone=GPT-4o-mini, Topology=distinct-shard2026.05 | 70.1 | — | |
| CrewAIN=16, Backbone=GPT-4o-mini, Topology=distinct-shard2026.05 | 41.6 | — | |
| CrewAIN=8, Backbone=GPT-4o-mini, Topology=distinct-shard2026.05 | 41.2 | — | |
| CrewAIN=4, Backbone=GPT-4o-mini, Topology=distinct-shard2026.05 | 39.8 | — |