Software Engineering Task Resolution on SWE-bench Verified
73.3Resolution RateTACT-GATE
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| TACT-GATEBackbone=Qwen3.5-27B2026.05 | 73.3 | — | 65.3 | |
| TACT-ADDBackbone=Qwen3.5-27B2026.05 | 73 | — | 67.5 | |
| AGENTDEBUGBackbone=Qwen3.5-27B, rollouts=at least two per task2026.05 | 70.5 | — | 116.4 | |
| TACT-CAPBackbone=Qwen3.5-27B2026.05 | 69.1 | — | 55.7 | |
| AGENTREFLECTBackbone=Qwen3.5-27B2026.05 | 68.4 | — | 73.1 | |
| RECAPBackbone=Qwen3.5-27B2026.05 | 67.9 | — | 75.8 | |
| Base ModelBackbone=Qwen3.5-27B2026.05 | 67 | — | 75.2 | |
| REMINDERBackbone=Qwen3.5-27B2026.05 | 66.2 | — | 78 | |
| TACT-GATEBackbone=Gemma-4-26B-A4B-it2026.05 | 65.5 | — | 44.7 | |
| TACT-ADDBackbone=Gemma-4-26B-A4B-it2026.05 | 65.2 | — | 46.2 | |
| AGENTDEBUGBackbone=Gemma-4-26B-A4B-it, rollouts=at least two per task2026.05 | 62 | — | 78.9 | |
| TACT-CAPBackbone=Gemma-4-26B-A4B-it2026.05 | 61.6 | — | 38.6 | |
| RECAPBackbone=Gemma-4-26B-A4B-it2026.05 | 61 | — | 52.5 | |
| AGENTREFLECTBackbone=Gemma-4-26B-A4B-it2026.05 | 60.4 | — | 51 | |
| REMINDERBackbone=Gemma-4-26B-A4B-it2026.05 | 60 | — | 53.2 | |
| Base ModelBackbone=Gemma-4-26B-A4B-it2026.05 | 59.7 | — | 52.1 | |
| ReasoningBankBackbone LLM=Gemini-2.5-pro2025.09 | 57.4 | 19.8 | — | |
| No MemoryBackbone LLM=Gemini-2.5-pro2025.09 | 54 | 21.1 | — | |
| SynapseBackbone LLM=Gemini-2.5-pro2025.09 | 53.4 | 21 | — | |
| ReasoningBankBackbone LLM=Gemini-2.5-flash2025.09 | 38.8 | 27.5 | — | |
| SWE-Dev-32BBackbone=~32B backbones, Scaffold=SWE-Dev, Data=SWE-Dev2026.05 | 36.6 | — | — | |
| SynapseBackbone LLM=Gemini-2.5-flash2025.09 | 35.4 | 30.7 | — | |
| R2E-Gym-32B-AgentBackbone=~32B backbones, Scaffold=OpenHands, Data=R2E-Gym2026.05 | 34.4 | — | — | |
| No MemoryBackbone LLM=Gemini-2.5-flash2025.09 | 34.2 | 30.3 | — | |
| SRFTTraining Data=SRFT2026.05 | 32.2 | — | — | |
| RFTTraining Data=RFT2026.05 | 30.9 | — | — | |
| DAgger-styleBackbone=Qwen3-8B, Scaffold=OpenHands, Data=SWE-Gym2026.05 | 29.8 | — | — | |
| Naïve distillationTraining Data=Naïve distillation2026.05 | 28.5 | — | — | |
| DAgger-styleBackbone=Qwen3-4B, Scaffold=OpenHands, Data=SWE-Gym2026.05 | 27.3 | — | — | |
| AggreVaTe-styleBackbone=Qwen3-8B, Scaffold=OpenHands, Data=SWE-Gym2026.05 | 27.3 | — | — | |
| Qwen3-8B + OPDBackbone=Qwen3-8B, Scaffold=OpenHands, Data=SWE-Gym2026.05 | 26.2 | — | — | |
| AggreVaTe-styleBackbone=Qwen3-4B, Scaffold=OpenHands, Data=SWE-Gym2026.05 | 24.5 | — | — | |
| Qwen3-4B + OPDBackbone=Qwen3-4B, Scaffold=OpenHands, Data=SWE-Gym2026.05 | 23.4 | — | — | |
| Qwen3-8B + SFTBackbone=Qwen3-8B, Scaffold=OpenHands, Data=SWE-Gym2026.05 | 23.4 | — | — | |
| Qwen3-4B + SFTBackbone=Qwen3-4B, Scaffold=OpenHands, Data=SWE-Gym2026.05 | 22.9 | — | — | |
| SWE-Gym-32BBackbone=~32B backbones, Scaffold=OpenHands, Data=SWE-Gym2026.05 | 20.6 | — | — | |
| R2E-Gym-7B-AgentBackbone=~7–8B backbones, Scaffold=OpenHands, Data=R2E-Gym2026.05 | 19 | — | — | |
| SWE-smith-LM-7BBackbone=~7–8B backbones, Scaffold=SWE-Agent, Data=SWE-smith2026.05 | 15.2 | — | — | |
| SkyRL-Agent-7B-v0Backbone=~7–8B backbones, Scaffold=OpenHands, Data=SkyRL-v02026.05 | 14.6 | — | — | |
| Qwen3-4B + GRPOBackbone=Qwen3-4B, Scaffold=OpenHands, Data=SkyRL-v02026.05 | 11.6 | — | — | |
| Qwen3-4BBackbone=Qwen3-4B, Scaffold=OpenHands, Data=–2026.05 | 11.2 | — | — | |
| SWE-Gym-7BBackbone=~7–8B backbones, Scaffold=OpenHands, Data=SWE-Gym2026.05 | 10.6 | — | — | |
| SkyRL-Agent-8B-v0Backbone=~7–8B backbones, Scaffold=OpenHands, Data=SkyRL-v02026.05 | 9.4 | — | — | |
| Qwen3-8B + GRPOBackbone=Qwen3-8B, Scaffold=OpenHands, Data=SkyRL-v02026.05 | 8.2 | — | — | |
| Qwen3-8BBackbone=Qwen3-8B, Scaffold=OpenHands, Data=–2026.05 | 7.7 | — | — | |
| Base modelTraining Data=Base model2026.05 | 7 | — | — | |
| Claude 4.0 SonnetScale=Large, Scaffold=OpenHands2025.12 | 0.704 | — | — | |
| Qwen3-Coder 480B/A35B InstructScale=Large, Scaffold=OpenHands2025.12 | 0.696 | — | — | |
| Claude 4.0 SonnetScale=Large, Scaffold=SWE-agent2025.12 | 0.668 | — | — | |
| Claude 3.7 SonnetScale=Large, Scaffold=SWE-agent2025.12 | 0.624 | — | — | |
| GLM-4.5-AirScale=Large, Scaffold=OpenHands2025.12 | 0.576 | — | — | |
| Seed-OSS-36B-InstructScale=Small, Scaffold=SWE-agent + BOAD2025.12 | 0.532 | — | — | |
| Claude 3.5 SonnetScale=Large, Scaffold=OpenHands2025.12 | 0.53 | — | — | |
| Qwen3-Coder-30B-A3B-InstructScale=Small, Scaffold=OpenHands2025.12 | 0.516 | — | — | |
| Claude 3.5 SonnetScale=Large, Scaffold=Agentless2025.12 | 0.508 | — | — | |
| Seed-OSS-36B-InstructScale=Small, Scaffold=SWE-agent (baseline)2025.12 | 0.498 | — | — | |
| DeepSeek-R1Scale=Large, Scaffold=Agentless2025.12 | 0.492 | — | — | |
| Seed-OSS-36B-InstructScale=Small, Scaffold=SWE-agent + Manual Sub-agent2025.12 | 0.474 | — | — | |
| Devstral-Small-2505Scale=Small, Scaffold=OpenHands2025.12 | 0.468 | — | — | |
| Seed-OSS-36B-InstructScale=Small, Scaffold=SWE-agent + Evolutionary Search2025.12 | 0.46 | — | — | |
| DeepSeek-V3Scale=Large, Scaffold=Agentless2025.12 | 0.42 | — | — | |
| GPT-4oScale=Large, Scaffold=Agentless2025.12 | 0.388 | — | — | |
| GPT-4oScale=Large, Scaffold=SWE-agent2025.12 | 0.23 | — | — |