Software Engineering on SWE-bench Verified (Accuracy)
62.6AccuracyJoyAI-LLM Flash
Evaluation Results
| Method | Links | |
|---|---|---|
| JoyAI-LLM Flash#Token=244002026.04 | 62.6 | |
| PivotRLBase Model=Nemotron-3-Super, Post-training Stage=After RL Stage2026.03 | 61.33 | |
| GLM-4.7-Flash-T#Token=514002026.04 | 59.4 | |
| Qwen3.5-35B-A3B#Token=232002026.04 | 57.4 | |
| AdaptOrchLat.=1.6×, Tok(K)=41.82026.02 | 52.6 | |
| Self-MoA (matched)Lat.=1.5×, Tok(K)=43.22026.02 | 51.5 | |
| MoA-3LLat.=3.2×, Tok(K)=84.62026.02 | 48.1 | |
| Static-ParallelLat.=1.4×, Tok(K)=52.12026.02 | 47.3 | |
| Static-SequentialLat.=2.8×, Tok(K)=48.92026.02 | 45.6 | |
| LLM-BlenderLat.=1.8×, Tok(K)=61.72026.02 | 44.9 | |
| Single BestLat.=1.0×, Tok(K)=12.32026.02 | 42.8 | |
| SWE-AgentModel=Qwen-2.5-32B-Coder-Instruct, Training Data=ADP Data2025.10 | 40.3 | |
| SWE-AgentModel=Qwen-2.5-32B-Coder-Instruct, Training Data=SWE-smith (Yang et al., 2025b)2025.10 | 40.2 | |
| OT-Agent-SFT-8B (100K)Base Model=OpenThinker-Agent-v1-SFT, Train Size=100K, Training Method=SFT2026.06 | 38.9 | |
| SWE-Lego-Qwen3-8BBase Model=Qwen3-8B, Train Size=18K, Training Method=SFT2026.06 | 37.6 | |
| OpenHands CodeActAgentModel=Qwen-2.5-32B-Coder-Instruct, Training Data=ADP Data2025.10 | 36.8 | |
| SWE-AgentModel=Qwen-2.5-14B-Coder-Instruct, Training Data=ADP Data2025.10 | 34.4 | |
| SWE-AgentModel=Claude 3.5 Sonnet(Anthropic Team), Training Data=–2025.10 | 33.6 | |
| OT-Agent-ColdSFT+RL-8BBase Model=OpenThinker-Agent-v1-SFT, Train Size=10K+5K, Training Method=SFT+RL2026.06 | 31.9 | |
| Qwen3-Next-80B-A3B#Token=254002026.04 | 31.2 | |
| OpenHands CodeActAgentModel=Qwen-2.5-14B-Coder-Instruct, Training Data=ADP Data2025.10 | 30.6 | |
| Qwen3-30B-A3B + SAOScaffold=OpenHands, Interaction turns=300, Context budget=128k-token2026.07 | 29.8 | |
| Qwen3-30B-A3B#Token=164002026.04 | 29 | |
| Qwen3-30B-A3B + GRPO (w/ DIS)Scaffold=OpenHands, Interaction turns=300, Context budget=128k-token2026.07 | 27 | |
| Qwen3-30B-A3BScaffold=OpenHands, Interaction turns=300, Context budget=128k-token2026.07 | 23 | |
| OT-Agent-SFT-8B (10K)Base Model=OpenThinker-Agent-v1-SFT, Train Size=10K, Training Method=SFT2026.06 | 22.7 | |
| Nemotron-Terminal-8BTrain Size=264K, Training Method=SFT2026.06 | 22.1 | |
| OpenHands CodeActAgentModel=Qwen-2.5-32B-Coder-Instruct, Training Data=SWE-Gym (Pan et al., 2025)2025.10 | 20.6 | |
| OpenHands CodeActAgentModel=Qwen-2.5-7B-Coder-Instruct, Training Data=ADP Data2025.10 | 20.4 | |
| SWE-AgentModel=Qwen-2.5-7B-Coder-Instruct, Training Data=ADP Data2025.10 | 20.2 | |
| OpenHands CodeActAgentModel=Qwen-2.5-14B-Coder-Instruct, Training Data=SWE-Gym (Pan et al., 2025)2025.10 | 16.4 | |
| SWE-AgentModel=Claude 3 Opus (Anthropic Team), Training Data=–2025.10 | 15.8 | |
| SWE-AgentModel=Qwen-2.5-7B-Coder-Instruct, Training Data=SWE-smith (Yang et al., 2025b)2025.10 | 15.2 | |
| Qwen3-8BBase Model=N/A, Train Size=N/A, Training Method=N/A2026.06 | 13.2 | |
| Nemotron-3-SuperPost-training Stage=SFT2026.03 | 12.87 | |
| Endless TerminalsBase Model=Qwen3-8B, Train Size=15K+3K, Training Method=SFT+RL2026.06 | 12.5 | |
| OpenHands CodeActAgentModel=Qwen-2.5-32B-Coder-Instruct, Training Data=–2025.10 | 10.6 | |
| OpenHands CodeActAgentModel=Qwen-2.5-7B-Coder-Instruct, Training Data=SWE-Gym (Pan et al., 2025)2025.10 | 10.6 | |
| OpenHands CodeActAgentModel=Qwen-2.5-14B-Coder-Instruct, Training Data=–2025.10 | 5.8 | |
| OpenHands CodeActAgentModel=Qwen-2.5-7B-Coder-Instruct, Training Data=–2025.10 | 2.8 | |
| SWE-AgentModel=Qwen-2.5-32B-Coder-Instruct, Training Data=–2025.10 | 2.2 | |
| SWE-AgentModel=Qwen-2.5-14B-Coder-Instruct, Training Data=–2025.10 | 2 | |
| SWE-AgentModel=Qwen-2.5-7B-Coder-Instruct, Training Data=–2025.10 | 0.4 |