Feature Implementation on FEA-Bench (test)
5.92Pass RateSWE-SPOT
Evaluation results
| Method | Links | ||
|---|---|---|---|
| SWE-SPOTSize=4B, SWE Training Data=8K trajs in one target repo2026.01 | 5.92 | 17.12 | |
| GPT-4.1-miniSize=[10, 100]B, SWE Training Data=-2026.01 | 5.7 | 17.85 | |
| Mini-Coder-4BSize=4B, SWE Training Data=400K trajs from 128 repos2026.01 | 4.61 | 8.7 | |
| CWM (Meta)Size=32B, SWE Training Data=3M trajs from 3.15K repos2026.01 | 4.17 | 15.88 | |
| Qwen3-Coder-30BSize=32B, SWE Training Data=-2026.01 | 3.29 | 11.56 | |
| GPT-5-nanoSize=[0.1, 10]B, SWE Training Data=-2026.01 | 1.75 | 2.97 | |
| Gemini-2.5-Flash-LiteSize=[0.1, 10]B, SWE Training Data=-2026.01 | 0.88 | 7.46 | |
| Gemma-3-27b-itSize=27B, SWE Training Data=-2026.01 | 0 | 1.34 | |
| Qwen3-4B-Instruct-2507Size=4B, SWE Training Data=-2026.01 | 0 | 3.2 |