Code Localization on SWE-Bench Lite
84.33File-Level RecallSHERLOC
Evaluation results
| Method | Links | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| SHERLOCBackbone=Qwen3-235B-A22B-Thinking2026.06 | 84.33 | 81.1 | 82.14 | 78.08 | 39.14 | 27.52 | 44.54 | 5 | 0 | |
| SHERLOCBackbone=DeepSeek-V3-03242026.06 | 79.67 | 68.2 | 71.81 | 58 | 49.26 | 10.63 | 46.67 | 4.4 | 3.3 | |
| SHERLOCBackbone=Qwen3-30B-A3B-Thinking2026.06 | 76.33 | 74.59 | 75.15 | 73 | 25.53 | 22.96 | 31.13 | 7.4 | 0.1 | |
| SHERLOCBackbone=DeepSeek-R1-05282026.06 | 75.78 | 71.45 | 72.72 | 68 | 36.49 | 20.72 | 39.44 | 3.4 | 12 |