File-level Code Localization on SWE-bench Verified Lite
91.9Accuracy@1Repo-Enc
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| Repo-EncModel=GPT-52026.02 | 91.9 | 97.7 | 91.1 | 89.1 | |
| Repo-EncModel=Claude-4.5-Sonnet2026.02 | 90.5 | 97.6 | 91.8 | 88.6 | |
| OrcaLocaModel=GPT-52026.02 | 88.2 | 93.9 | 88.6 | 84.2 | |
| LocAgentModel=GPT-52026.02 | 88.2 | 96.7 | 88.4 | 86.7 | |
| OrcaLocaModel=Claude-4.5-Sonnet2026.02 | 87.2 | 89.6 | 87.5 | 82.2 | |
| CoSILModel=GPT-52026.02 | 82.8 | 95.7 | 82.3 | 80.2 | |
| Repo-EncModel=GPT-4.12026.02 | 82.6 | 93.2 | 83.6 | 79.3 | |
| LocAgentModel=GPT-4.12026.02 | 79.5 | 90.9 | 80.8 | 77.2 | |
| AgentlessModel=GPT-52026.02 | 78.7 | 95.9 | 78.3 | 76.2 | |
| Repo-EncModel=o3-mini2026.02 | 78.3 | 91.2 | 80.7 | 76.8 | |
| AgentlessModel=Claude-4.5-Sonnet2026.02 | 76.6 | 96.5 | 76.9 | 74.4 | |
| CoSILModel=Claude-4.5-Sonnet2026.02 | 75.5 | 96.1 | 75.9 | 73.7 | |
| OrcaLocaModel=GPT-4.12026.02 | 75.2 | 80 | 76.5 | 71.3 | |
| Repo-EncModel=GPT-4o2026.02 | 74.5 | 89.6 | 77 | 72.7 | |
| LocAgentModel=GPT-4o2026.02 | 71.9 | 87.9 | 73.4 | 69.3 | |
| LocAgentModel=Claude-4.5-Sonnet2026.02 | 71.4 | 76.6 | 72.7 | 70.2 | |
| CoSILModel=GPT-4.12026.02 | 69.8 | 90.6 | 70.7 | 67.6 | |
| OrcaLocaModel=o3-mini2026.02 | 67.5 | 71.9 | 68.3 | 64 | |
| AgentlessModel=o3-mini2026.02 | 67.1 | 88.1 | 67 | 64.7 | |
| CoSILModel=o3-mini2026.02 | 66.5 | 85.7 | 66.2 | 63.6 | |
| AgentlessModel=GPT-4.12026.02 | 65.2 | 90.8 | 65.7 | 63.5 | |
| CoSILModel=GPT-4o2026.02 | 64.9 | 84.4 | 65 | 62.2 | |
| OrcaLocaModel=GPT-4o2026.02 | 64.3 | 69.3 | 65 | 61.4 | |
| AgentlessModel=GPT-4o2026.02 | 63 | 86.1 | 63.1 | 61.1 | |
| LocAgentModel=o3-mini2026.02 | 62.8 | 77.2 | 64.7 | 61.4 |