List your company
ResearchFormerly WizWand

Software Engineering on SWE-bench Verified (Accuracy)

62.6AccuracyJoyAI-LLM Flash
-2.08814.70631.548.294Oct 28, 2025Dec 9, 2025Jan 20, 2026Mar 3, 2026Apr 14, 2026May 26, 2026Jul 8, 2026
Updated 3mo ago

Evaluation results

MethodLinks
JoyAI-LLM Flash#Token=244002026.0462.6
PivotRLBase Model=Nemotron-3-Super, Post-training Stage=After RL Stage2026.0361.33
GLM-4.7-Flash-T#Token=514002026.0459.4
Qwen3.5-35B-A3B#Token=232002026.0457.4
AdaptOrchLat.=1.6×, Tok(K)=41.82026.0252.6
Self-MoA (matched)Lat.=1.5×, Tok(K)=43.22026.0251.5
MoA-3LLat.=3.2×, Tok(K)=84.62026.0248.1
Static-ParallelLat.=1.4×, Tok(K)=52.12026.0247.3
Static-SequentialLat.=2.8×, Tok(K)=48.92026.0245.6
LLM-BlenderLat.=1.8×, Tok(K)=61.72026.0244.9
Single BestLat.=1.0×, Tok(K)=12.32026.0242.8
SWE-AgentModel=Qwen-2.5-32B-Coder-Instruct, Training Data=ADP Data2025.1040.3
SWE-AgentModel=Qwen-2.5-32B-Coder-Instruct, Training Data=SWE-smith (Yang et al., 2025b)2025.1040.2
OT-Agent-SFT-8B (100K)Base Model=OpenThinker-Agent-v1-SFT, Train Size=100K, Training Method=SFT2026.0638.9
SWE-Lego-Qwen3-8BBase Model=Qwen3-8B, Train Size=18K, Training Method=SFT2026.0637.6
OpenHands CodeActAgentModel=Qwen-2.5-32B-Coder-Instruct, Training Data=ADP Data2025.1036.8
SWE-AgentModel=Qwen-2.5-14B-Coder-Instruct, Training Data=ADP Data2025.1034.4
SWE-AgentModel=Claude 3.5 Sonnet(Anthropic Team), Training Data=–2025.1033.6
OT-Agent-ColdSFT+RL-8BBase Model=OpenThinker-Agent-v1-SFT, Train Size=10K+5K, Training Method=SFT+RL2026.0631.9
Qwen3-Next-80B-A3B#Token=254002026.0431.2
OpenHands CodeActAgentModel=Qwen-2.5-14B-Coder-Instruct, Training Data=ADP Data2025.1030.6
Qwen3-30B-A3B + SAOScaffold=OpenHands, Interaction turns=300, Context budget=128k-token2026.0729.8
Qwen3-30B-A3B#Token=164002026.0429
Qwen3-30B-A3B + GRPO (w/ DIS)Scaffold=OpenHands, Interaction turns=300, Context budget=128k-token2026.0727
Qwen3-30B-A3BScaffold=OpenHands, Interaction turns=300, Context budget=128k-token2026.0723
OT-Agent-SFT-8B (10K)Base Model=OpenThinker-Agent-v1-SFT, Train Size=10K, Training Method=SFT2026.0622.7
Nemotron-Terminal-8BTrain Size=264K, Training Method=SFT2026.0622.1
OpenHands CodeActAgentModel=Qwen-2.5-32B-Coder-Instruct, Training Data=SWE-Gym (Pan et al., 2025)2025.1020.6
OpenHands CodeActAgentModel=Qwen-2.5-7B-Coder-Instruct, Training Data=ADP Data2025.1020.4
SWE-AgentModel=Qwen-2.5-7B-Coder-Instruct, Training Data=ADP Data2025.1020.2
OpenHands CodeActAgentModel=Qwen-2.5-14B-Coder-Instruct, Training Data=SWE-Gym (Pan et al., 2025)2025.1016.4
SWE-AgentModel=Claude 3 Opus (Anthropic Team), Training Data=–2025.1015.8
SWE-AgentModel=Qwen-2.5-7B-Coder-Instruct, Training Data=SWE-smith (Yang et al., 2025b)2025.1015.2
Qwen3-8BBase Model=N/A, Train Size=N/A, Training Method=N/A2026.0613.2
Nemotron-3-SuperPost-training Stage=SFT2026.0312.87
Endless TerminalsBase Model=Qwen3-8B, Train Size=15K+3K, Training Method=SFT+RL2026.0612.5
OpenHands CodeActAgentModel=Qwen-2.5-32B-Coder-Instruct, Training Data=–2025.1010.6
OpenHands CodeActAgentModel=Qwen-2.5-7B-Coder-Instruct, Training Data=SWE-Gym (Pan et al., 2025)2025.1010.6
OpenHands CodeActAgentModel=Qwen-2.5-14B-Coder-Instruct, Training Data=–2025.105.8
OpenHands CodeActAgentModel=Qwen-2.5-7B-Coder-Instruct, Training Data=–2025.102.8
SWE-AgentModel=Qwen-2.5-32B-Coder-Instruct, Training Data=–2025.102.2
SWE-AgentModel=Qwen-2.5-14B-Coder-Instruct, Training Data=–2025.102
SWE-AgentModel=Qwen-2.5-7B-Coder-Instruct, Training Data=–2025.100.4