Test Generation on TDD-Bench Verified (test)
22.75Pass RateSWE-SPOT
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| SWE-SPOTSize=4B, SWE Training Data=8K trajs in one target repo2026.01 | 22.75 | 17.12 | |
| GPT-4.1-miniSize=[10, 100]B, SWE Training Data=-2026.01 | 22.27 | 17.85 | |
| CWM (Meta)Size=32B, SWE Training Data=3M trajs from 3.15K repos2026.01 | 17.38 | 15.88 | |
| Qwen3-Coder-30BSize=32B, SWE Training Data=-2026.01 | 11.85 | 11.56 | |
| Gemini-2.5-Flash-LiteSize=[0.1, 10]B, SWE Training Data=-2026.01 | 10.74 | 7.46 | |
| Qwen3-4B-Instruct-2507Size=4B, SWE Training Data=-2026.01 | 5.37 | 3.2 | |
| Gemma-3-27b-itSize=27B, SWE Training Data=-2026.01 | 2.37 | 1.34 | |
| Mini-Coder-4BSize=4B, SWE Training Data=400K trajs from 128 repos2026.01 | 0.63 | 8.7 | |
| GPT-5-nanoSize=[0.1, 10]B, SWE Training Data=-2026.01 | 0 | 2.97 |