List your company
ResearchFormerly WizWand

Software Engineering Task Resolution on SWE-bench Verified

73.3Resolution RateTACT-GATE
-2.692817.036136.76556.4939Sep 29, 2025Nov 5, 2025Dec 13, 2025Jan 20, 2026Feb 26, 2026Apr 5, 2026May 13, 2026
Updated 5mo ago

Evaluation results

MethodLinks
TACT-GATEBackbone=Qwen3.5-27B2026.0573.3—65.3
TACT-ADDBackbone=Qwen3.5-27B2026.0573—67.5
AGENTDEBUGBackbone=Qwen3.5-27B, rollouts=at least two per task2026.0570.5—116.4
TACT-CAPBackbone=Qwen3.5-27B2026.0569.1—55.7
AGENTREFLECTBackbone=Qwen3.5-27B2026.0568.4—73.1
RECAPBackbone=Qwen3.5-27B2026.0567.9—75.8
Base ModelBackbone=Qwen3.5-27B2026.0567—75.2
REMINDERBackbone=Qwen3.5-27B2026.0566.2—78
TACT-GATEBackbone=Gemma-4-26B-A4B-it2026.0565.5—44.7
TACT-ADDBackbone=Gemma-4-26B-A4B-it2026.0565.2—46.2
AGENTDEBUGBackbone=Gemma-4-26B-A4B-it, rollouts=at least two per task2026.0562—78.9
TACT-CAPBackbone=Gemma-4-26B-A4B-it2026.0561.6—38.6
RECAPBackbone=Gemma-4-26B-A4B-it2026.0561—52.5
AGENTREFLECTBackbone=Gemma-4-26B-A4B-it2026.0560.4—51
REMINDERBackbone=Gemma-4-26B-A4B-it2026.0560—53.2
Base ModelBackbone=Gemma-4-26B-A4B-it2026.0559.7—52.1
ReasoningBankBackbone LLM=Gemini-2.5-pro2025.0957.419.8—
No MemoryBackbone LLM=Gemini-2.5-pro2025.095421.1—
SynapseBackbone LLM=Gemini-2.5-pro2025.0953.421—
ReasoningBankBackbone LLM=Gemini-2.5-flash2025.0938.827.5—
SWE-Dev-32BBackbone=~32B backbones, Scaffold=SWE-Dev, Data=SWE-Dev2026.0536.6——
SynapseBackbone LLM=Gemini-2.5-flash2025.0935.430.7—
R2E-Gym-32B-AgentBackbone=~32B backbones, Scaffold=OpenHands, Data=R2E-Gym2026.0534.4——
No MemoryBackbone LLM=Gemini-2.5-flash2025.0934.230.3—
SRFTTraining Data=SRFT2026.0532.2——
RFTTraining Data=RFT2026.0530.9——
DAgger-styleBackbone=Qwen3-8B, Scaffold=OpenHands, Data=SWE-Gym2026.0529.8——
Naïve distillationTraining Data=Naïve distillation2026.0528.5——
DAgger-styleBackbone=Qwen3-4B, Scaffold=OpenHands, Data=SWE-Gym2026.0527.3——
AggreVaTe-styleBackbone=Qwen3-8B, Scaffold=OpenHands, Data=SWE-Gym2026.0527.3——
Qwen3-8B + OPDBackbone=Qwen3-8B, Scaffold=OpenHands, Data=SWE-Gym2026.0526.2——
AggreVaTe-styleBackbone=Qwen3-4B, Scaffold=OpenHands, Data=SWE-Gym2026.0524.5——
Qwen3-4B + OPDBackbone=Qwen3-4B, Scaffold=OpenHands, Data=SWE-Gym2026.0523.4——
Qwen3-8B + SFTBackbone=Qwen3-8B, Scaffold=OpenHands, Data=SWE-Gym2026.0523.4——
Qwen3-4B + SFTBackbone=Qwen3-4B, Scaffold=OpenHands, Data=SWE-Gym2026.0522.9——
SWE-Gym-32BBackbone=~32B backbones, Scaffold=OpenHands, Data=SWE-Gym2026.0520.6——
R2E-Gym-7B-AgentBackbone=~7–8B backbones, Scaffold=OpenHands, Data=R2E-Gym2026.0519——
SWE-smith-LM-7BBackbone=~7–8B backbones, Scaffold=SWE-Agent, Data=SWE-smith2026.0515.2——
SkyRL-Agent-7B-v0Backbone=~7–8B backbones, Scaffold=OpenHands, Data=SkyRL-v02026.0514.6——
Qwen3-4B + GRPOBackbone=Qwen3-4B, Scaffold=OpenHands, Data=SkyRL-v02026.0511.6——
Qwen3-4BBackbone=Qwen3-4B, Scaffold=OpenHands, Data=–2026.0511.2——
SWE-Gym-7BBackbone=~7–8B backbones, Scaffold=OpenHands, Data=SWE-Gym2026.0510.6——
SkyRL-Agent-8B-v0Backbone=~7–8B backbones, Scaffold=OpenHands, Data=SkyRL-v02026.059.4——
Qwen3-8B + GRPOBackbone=Qwen3-8B, Scaffold=OpenHands, Data=SkyRL-v02026.058.2——
Qwen3-8BBackbone=Qwen3-8B, Scaffold=OpenHands, Data=–2026.057.7——
Base modelTraining Data=Base model2026.057——
Claude 4.0 SonnetScale=Large, Scaffold=OpenHands2025.120.704——
Qwen3-Coder 480B/A35B InstructScale=Large, Scaffold=OpenHands2025.120.696——
Claude 4.0 SonnetScale=Large, Scaffold=SWE-agent2025.120.668——
Claude 3.7 SonnetScale=Large, Scaffold=SWE-agent2025.120.624——
GLM-4.5-AirScale=Large, Scaffold=OpenHands2025.120.576——
Seed-OSS-36B-InstructScale=Small, Scaffold=SWE-agent + BOAD2025.120.532——
Claude 3.5 SonnetScale=Large, Scaffold=OpenHands2025.120.53——
Qwen3-Coder-30B-A3B-InstructScale=Small, Scaffold=OpenHands2025.120.516——
Claude 3.5 SonnetScale=Large, Scaffold=Agentless2025.120.508——
Seed-OSS-36B-InstructScale=Small, Scaffold=SWE-agent (baseline)2025.120.498——
DeepSeek-R1Scale=Large, Scaffold=Agentless2025.120.492——
Seed-OSS-36B-InstructScale=Small, Scaffold=SWE-agent + Manual Sub-agent2025.120.474——
Devstral-Small-2505Scale=Small, Scaffold=OpenHands2025.120.468——
Seed-OSS-36B-InstructScale=Small, Scaffold=SWE-agent + Evolutionary Search2025.120.46——
DeepSeek-V3Scale=Large, Scaffold=Agentless2025.120.42——
GPT-4oScale=Large, Scaffold=Agentless2025.120.388——
GPT-4oScale=Large, Scaffold=SWE-agent2025.120.23——