Code Localization on SWE-Bench Verified
81.27File-Level RecallSHERLOC
Evaluation results
| Method | Links | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| SHERLOCBackbone=Qwen3-235B-A22B-Thinking2026.06 | 81.27 | 87.55 | 83.53 | 75.2 | 41.28 | 36.36 | 53.47 | 4.7 | 0 | |
| SHERLOCBackbone=DeepSeek-V3-03242026.06 | 79.53 | 75.47 | 75.46 | 60.2 | 54.4 | 14.5 | 56.4 | 4.7 | 0 | |
| SHERLOCBackbone=Qwen3-30B-A3B-Thinking2026.06 | 75.07 | 78.5 | 74.96 | 67.8 | 29.8 | 29.4 | 39.4 | 7.2 | 0 | |
| SHERLOCBackbone=DeepSeek-R1-05282026.06 | 73.63 | 76.12 | 73.39 | 65.2 | 37.7 | 25.7 | 45.9 | 3.2 | 10 |