Codebase QA on SWE-QA (test)
80.28ScoreGPT-4.1-mini
Evaluation results
| Method | Links | |
|---|---|---|
| GPT-4.1-miniSize=[10, 100]B, SWE Training Data=-2026.01 | 80.28 | |
| SWE-SPOTSize=4B, SWE Training Data=8K trajs in one target repo2026.01 | 78.05 | |
| Gemini-2.5-Flash-LiteSize=[0.1, 10]B, SWE Training Data=-2026.01 | 73.36 | |
| CWM (Meta)Size=32B, SWE Training Data=3M trajs from 3.15K repos2026.01 | 73.09 | |
| Qwen3-Coder-30BSize=32B, SWE Training Data=-2026.01 | 65.48 | |
| Gemma-3-27b-itSize=27B, SWE Training Data=-2026.01 | 65.19 | |
| Qwen3-4B-Instruct-2507Size=4B, SWE Training Data=-2026.01 | 61.79 | |
| Mini-Coder-4BSize=4B, SWE Training Data=400K trajs from 128 repos2026.01 | 57.3 | |
| GPT-5-nanoSize=[0.1, 10]B, SWE Training Data=-2026.01 | 35.64 |