Autonomous Software Engineering on SWE-Bench-Verified (test)
69Resolution Rate (%)Full-Context
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| Full-ContextAgent Backbone=GLM-4.7 (355B 32B), Memory Strategy=Full-Context2026.05 | 69 | 55.11 | 6,361.49 | 1 | 5,869.42 | 1 | |
| CoMem (GRPO_AC)Agent Backbone=GLM-4.7 (355B 32B), Memory Strategy=GRPO_AC2026.05 | 62.7 | 51.21 | 3,318.42 | 1.92 | 2,821.38 | 2.08 | |
| Qwen3-4B (SFT)Agent Backbone=GLM-4.7 (355B 32B), Memory Strategy=Qwen3-4B (SFT)2026.05 | 61.3 | 52.74 | 4,087.92 | 1.56 | 3,668.53 | 1.6 | |
| No SummaryAgent Backbone=GLM-4.7 (355B 32B), Memory Strategy=No Summary2026.05 | 59.4 | 46.82 | 4,216.83 | 1.51 | 3,842.17 | 1.53 | |
| Qwen3-4B (base)Agent Backbone=GLM-4.7 (355B 32B), Memory Strategy=Qwen3-4B (base)2026.05 | 58.3 | 49.32 | 3,928.44 | 1.62 | 3,526.91 | 1.66 | |
| Full-ContextAgent Backbone=Qwen3-Coder-Max (480B A35B), Memory Strategy=Full-Context2026.05 | 57.2 | 36.68 | 6,291.11 | 1 | 5,129.9 | 1 | |
| CoMem (GRPO_AC)Agent Backbone=Qwen3-Coder-Max (480B A35B), Memory Strategy=GRPO_AC2026.05 | 51 | 44.55 | 3,917.68 | 1.61 | 3,594.51 | 1.43 | |
| Qwen3-4B (SFT)Agent Backbone=Qwen3-Coder-Max (480B A35B), Memory Strategy=Qwen3-4B (SFT)2026.05 | 47.3 | 43.39 | 3,819.7 | 1.65 | 3,337.58 | 1.54 | |
| No SummaryAgent Backbone=Qwen3-Coder-Max (480B A35B), Memory Strategy=No Summary2026.05 | 46.6 | 25.2 | 3,780.74 | 1.66 | 3,421.78 | 1.5 | |
| Qwen3-4B (base)Agent Backbone=Qwen3-Coder-Max (480B A35B), Memory Strategy=Qwen3-4B (base)2026.05 | 42.2 | 39.4 | 3,421.63 | 1.84 | 3,138.97 | 1.63 | |
| CoMem (GRPO_AC)Agent Backbone=DeepSWE (32B Dense), Memory Strategy=GRPO_AC2026.05 | 41 | 40.69 | 5,622.33 | 1.68 | 4,400.89 | 1.45 | |
| Full-ContextAgent Backbone=DeepSWE (32B Dense), Memory Strategy=Full-Context2026.05 | 40.4 | 34.39 | 9,457.78 | 1 | 6,390.62 | 1 | |
| Qwen3-4B (SFT)Agent Backbone=DeepSWE (32B Dense), Memory Strategy=Qwen3-4B (SFT)2026.05 | 39.8 | 35.61 | 5,232.64 | 1.81 | 4,168.85 | 1.53 | |
| Qwen3-4B (base)Agent Backbone=DeepSWE (32B Dense), Memory Strategy=Qwen3-4B (base)2026.05 | 29.9 | 24.74 | 4,360.72 | 2.17 | 3,649.61 | 1.75 |