Software Engineering on SWE-bench Verified (resolution rate)
83.8Resolution RateMemCoder
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| MemCoderSetting=pass@2, Backbone=GPT-5.22026.02 | 83.8 | — | 419 | |
| Claude Opus 4.6Scaffold=Claude Code2026.03 | 80.8 | — | — | |
| KAT-Coder-V2Scaffold=Claude Code2026.03 | 79.6 | — | — | |
| MemCoderSetting=pass@1, Backbone=GPT-5.22026.02 | 78.8 | — | 394 | |
| MemCoderSetting=pass@1, Backbone=DeepSeek-V3.22026.02 | 77.8 | — | 389 | |
| OpenHandsSetting=pass@3, Backbone=Claude Opus 4.52026.02 | 77.6 | — | 388 | |
| Claude Opus 4.6Scaffold=OpenClaw2026.03 | 75.7 | — | — | |
| Claude Opus 4.6Scaffold=OpenCode2026.03 | 75 | — | — | |
| KAT-Coder-V2Scaffold=OpenCode2026.03 | 74.8 | — | — | |
| OpenHandsSetting=pass@3, Backbone=Claude Sonnet 4.52026.02 | 74.6 | — | 373 | |
| OpenHandsSetting=pass@3, Backbone=GPT-5.22026.02 | 74.4 | — | 372 | |
| KAT-Coder-V2Scaffold=OpenClaw2026.03 | 72.8 | — | — | |
| OpenHandsSetting=pass@3, Backbone=Gemini 3 pro2026.02 | 70.4 | — | 352 | |
| Full-ContextAgent Backbone=GLM-4.7, Latency (s)=5869.42, Speedup=1.00×2026.05 | 69 | — | — | |
| COMEM (GRPO)Agent Backbone=GLM-4.7, Latency (s)=2821.38, Speedup=2.08×2026.05 | 62.7 | — | — | |
| Full-ContextAgent Backbone=Qwen3-Coder-Max, Latency (s)=5129.90, Speedup=1.00×2026.05 | 57.2 | — | — | |
| MemAgent (7B)Agent Backbone=GLM-4.7, Latency (s)=3253.83, Speedup=1.80×2026.05 | 54.8 | — | — | |
| COMEM (GRPO)Agent Backbone=Qwen3-Coder-Max, Latency (s)=3594.51, Speedup=1.43×2026.05 | 51 | — | — | |
| MemAgent (7B)Agent Backbone=Qwen3-Coder-Max, Latency (s)=3672.48, Speedup=1.40×2026.05 | 43.2 | — | — | |
| Proxy-filtered (code survival = 1)Selection Strategy=Proxy-filtered (code survival = 1)2026.03 | 0.504 | 3.8 | — | |
| Critic-selectedSelection Strategy=Critic-selected2026.03 | 0.478 | 1.2 | — | |
| Base Model (no SFT)Selection Strategy=Base Model (no SFT)2026.03 | 0.466 | — | — | |
| RandomSelection Strategy=Random2026.03 | 0.462 | -0.4 | — | |
| SWE-smith-LM-32BScaffold=SWE-Agent, Train size=5,016, Evaluation=Reported2025.12 | 0.402 | — | — | |
| OEC-SWE-32BScaffold=SWE-Agent, Train size=4,961, Evaluation=Own evaluation2025.12 | 0.4 | — | — | |
| Skywork-SWE-32BScaffold=OpenHands, Train size=8,4472025.12 | 0.38 | — | — | |
| SWE-smith-LM-32BScaffold=SWE-Agent, Train size=5,016, Evaluation=Own evaluation2025.12 | 0.364 | — | — | |
| OEC-SWE-7BScaffold=SWE-Agent, Train size=8,943, Evaluation=Own evaluation2025.12 | 0.208 | — | — | |
| SWE-Gym-32BScaffold=OpenHands, Train size=4912025.12 | 0.206 | — | — | |
| SWE-smith-LM-7BScaffold=SWE-Agent, Train size=5,0162025.12 | 0.152 | — | — | |
| SkyRL-Agent-7B-v0Scaffold=OpenHands, Train size=3,680+2025.12 | 0.146 | — | — | |
| SWE-Gym-7BScaffold=OpenHands, Train size=4912025.12 | 0.106 | — | — |