Automated Software Engineering on SWE-bench Verified
1,770Resolved RateYoutu-LLM 2B
Evaluation Results
| Method | Links | |
|---|---|---|
| Youtu-LLM 2BNon-thinking Mode=true, Scaffold=R2E-Gym/Openhands, Maximum Step=502025.12 | 1,770 | |
| SmolLM3 3BNon-thinking Mode=true, Scaffold=R2E-Gym/Openhands, Maximum Step=502025.12 | 720 | |
| Qwen3 4BNon-thinking Mode=true, Scaffold=R2E-Gym/Openhands, Maximum Step=502025.12 | 570 | |
| AIRBackbone=Gemini 2.52025.12 | 87.1 | |
| OpenHandsBackbone=GPT-52025.12 | 79.8 | |
| Devstral-Small-2Open Source Code=false, Open Source Model=true, Open Source Data=false, Base Model=24B, Teacher=—, Context=256K2026.01 | 68 | |
| SWE-agentBackbone=Claude 4 Sonnet2025.12 | 66.6 | |
| GLM-4.6Open Source Code=false, Open Source Model=true, Open Source Data=false, Base Model=357B, Teacher=—, Context=32K2026.01 | 60.8 | |
| Qwen3 1.7BNon-thinking Mode=true, Scaffold=R2E-Gym/Openhands, Maximum Step=502025.12 | 60 | |
| GLM-4.7-FlashOpen Source Code=false, Open Source Model=true, Open Source Data=false, Base Model=30B, Teacher=—, Context=128K2026.01 | 59.2 | |
| Devstral-Small-2Open Source Code=false, Open Source Model=true, Open Source Data=false, Base Model=24B, Teacher=—, Context=64K2026.01 | 59.1 | |
| GLM-4.5-AirOpen Source Code=false, Open Source Model=true, Open Source Data=false, Base Model=110B, Teacher=—, Context=64K2026.01 | 57.4 | |
| FrogBoss-32BOpen Source Code=false, Open Source Model=true, Open Source Data=false, Base Model=Qwen 3-32B, Teacher=Claude 4 Sonnet, Context=64K2026.01 | 54.6 | |
| SERA-32BOpen Source Code=true, Open Source Model=true, Open Source Data=true, Base Model=Qwen 3-32B, Teacher=GLM-4.6, Context=64K2026.01 | 54.2 | |
| CWMOpen Source Code=false, Open Source Model=true, Open Source Data=false, Base Model=32B, Teacher=—, Context=128K2026.01 | 53.9 | |
| OpenHandsBackbone=Claude 3.5 Sonnet2025.12 | 52.8 | |
| SERA-32B-GAOpen Source Code=true, Open Source Model=true, Open Source Data=true, Base Model=Qwen 3-32B, Teacher=GLM-4.5-Air, Context=64K2026.01 | 51.7 | |
| Qwen 3-Coder-30BOpen Source Code=false, Open Source Model=true, Open Source Data=false, Base Model=30B, Teacher=—, Context=256K2026.01 | 51.6 | |
| GLM-4.5-AirOpen Source Code=false, Open Source Model=true, Open Source Data=false, Base Model=110B, Teacher=—, Context=32K2026.01 | 50.5 | |
| Devstral-Small-2Open Source Code=false, Open Source Model=true, Open Source Data=false, Base Model=24B, Teacher=—, Context=32K2026.01 | 50 | |
| SERA-32BOpen Source Code=true, Open Source Model=true, Open Source Data=true, Base Model=Qwen 3-32B, Teacher=GLM-4.6, Context=32K2026.01 | 49.5 | |
| Kimi-devOpen Source Code=false, Open Source Model=true, Open Source Data=false, Base Model=72B, Teacher=—, Context=32K2026.01 | 48.6 | |
| SWE-agentBackbone=Claude 3.5 Sonnet2025.12 | 47.9 | |
| SERA-32B-GAOpen Source Code=true, Open Source Model=true, Open Source Data=true, Base Model=Qwen 3-32B, Teacher=GLM-4.5-Air, Context=32K2026.01 | 46.6 | |
| AutoCodeRoverBackbone=GPT-4o2025.12 | 46.2 | |
| Qwen 3-Coder-30BOpen Source Code=false, Open Source Model=true, Open Source Data=false, Base Model=30B, Teacher=—, Context=32K2026.01 | 45 | |
| SWE-SwissOpen Source Code=true, Open Source Model=true, Base Model=Qwen 2.5-32B, Teacher=—, Context=128K2026.01 | 45 | |
| DeepSWEOpen Source Code=true, Open Source Model=true, Base Model=Qwen 3-32B, Teacher=—, Context=32K2026.01 | 42.2 | |
| GLM-4.7-FlashOpen Source Code=false, Open Source Model=true, Open Source Data=false, Base Model=30B, Teacher=—, Context=64K2026.01 | 39.7 | |
| SkyRL-AgentOpen Source Code=true, Open Source Model=true, Base Model=Qwen 3-32B, Teacher=—, Context=32K2026.01 | 39.4 | |
| GLM-4.7-FlashOpen Source Code=false, Open Source Model=true, Open Source Data=false, Base Model=30B, Teacher=—, Context=32K2026.01 | 37.3 | |
| FrogBoss-32BOpen Source Code=false, Open Source Model=true, Open Source Data=false, Base Model=Qwen 3-32B, Teacher=Claude 4 Sonnet, Context=32K2026.01 | 35 | |
| SWE-smithOpen Source Code=true, Open Source Model=true, Open Source Data=true, Base Model=Qwen 2.5-32B, Teacher=Claude 3.7, Context=32K2026.01 | 32.6 | |
| SERA-8B-GAOpen Source Code=true, Open Source Model=true, Open Source Data=true, Base Model=Qwen 3-8B, Teacher=GLM-4.5-Air, Context=32K2026.01 | 31.7 | |
| SERA-8BOpen Source Code=true, Open Source Model=true, Open Source Data=true, Base Model=Qwen 3-8B, Teacher=GLM-4.6, Context=32K2026.01 | 31.7 | |
| SWE-smithBase Model=Qwen 3-32B, Teacher=Claude 3.7, Context=32K2026.01 | 25.6 | |
| Qwen 3-32BOpen Source Code=false, Open Source Model=true, Open Source Data=false, Base Model=32B, Teacher=—, Context=32K2026.01 | 24.4 | |
| Nex-N1-8BOpen Source Code=true, Open Source Model=true, Base Model=InternLM3-8B, Teacher=—, Context=32K2026.01 | 20.3 | |
| SkyRL-8BOpen Source Code=true, Open Source Model=true, Base Model=Qwen 3-8B, Teacher=—, Context=32K2026.01 | 9.4 |