Code Evaluation on SWE Verified Agentless
57.6pass@1DeepSeek-R1 0528 671B
Evaluation results
| Method | Links | |
|---|---|---|
| DeepSeek-R1 0528 671BParameters=671B, Thinking Mode=true, Setup=Agentless2025.12 | 57.6 | |
| Nemotron-Cascade 14B-Thinking (Test-Time Scaling)Parameters=14B, Thinking Mode=true, Test-Time Scaling=true2025.12 | 53.8 | |
| Gemini-2.5 Flash-ThinkingThinking Mode=true, Setup=Own scaffolds2025.12 | 48.9 | |
| Nemotron Cascade-8B (Test-Time Scaling)Parameters=8B, Thinking Mode=true, Test-Time Scaling=true2025.12 | 43.6 | |
| Nemotron-Cascade 14B-ThinkingParameters=14B, Thinking Mode=true, Setup=Agentless2025.12 | 43.1 | |
| Nemotron Cascade-8BParameters=8B, Thinking Mode=true, Setup=Agentless2025.12 | 37.2 | |
| Qwen3 14BParameters=14B, Setup=Agentless2025.12 | 27.4 | |
| Qwen3 8BParameters=8B, Setup=Agentless2025.12 | 20.5 |