Function-level Code Localization on SWE-bench Verified (Lite)
83.4Acc@1Repo-Enc
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| Repo-EncModel=GPT-52026.02 | 83.4 | 93.6 | 84.5 | 76.9 | |
| Repo-EncModel=Claude-4.5-Sonnet2026.02 | 79.8 | 93.7 | 83.4 | 75.8 | |
| OrcaLocaModel=GPT-52026.02 | 76.1 | 86.2 | 79.1 | 68.6 | |
| OrcaLocaModel=Claude-4.5-Sonnet2026.02 | 74.5 | 79.3 | 76.5 | 65.1 | |
| Repo-EncModel=GPT-4.12026.02 | 68.7 | 83.4 | 71 | 62.4 | |
| CoSILModel=GPT-52026.02 | 68.3 | 81.8 | 68.9 | 62.3 | |
| Repo-EncModel=o3-mini2026.02 | 58.5 | 77.8 | 62.9 | 55.1 | |
| CoSILModel=Claude-4.5-Sonnet2026.02 | 57.5 | 78.7 | 60.7 | 52.9 | |
| OrcaLocaModel=GPT-4.12026.02 | 55.2 | 66.7 | 59 | 50.1 | |
| Repo-EncModel=GPT-4o2026.02 | 53.1 | 76.7 | 57.9 | 49.5 | |
| CoSILModel=o3-mini2026.02 | 52.2 | 73.3 | 54.7 | 47.1 | |
| CoSILModel=GPT-4.12026.02 | 51.8 | 74.5 | 55.3 | 47 | |
| LocAgentModel=GPT-52026.02 | 50.9 | 80.3 | 55.9 | 49.7 | |
| LocAgentModel=Claude-4.5-Sonnet2026.02 | 49.3 | 57.8 | 51.5 | 44.9 | |
| OrcaLocaModel=o3-mini2026.02 | 46.3 | 52.9 | 48.3 | 41.5 | |
| AgentlessModel=GPT-52026.02 | 45.1 | 68.1 | 47.3 | 41.3 | |
| CoSILModel=GPT-4o2026.02 | 43.2 | 66.2 | 48.2 | 40.1 | |
| LocAgentModel=GPT-4o2026.02 | 40.1 | 67.4 | 44.8 | 38.1 | |
| OrcaLocaModel=GPT-4o2026.02 | 39.8 | 53.3 | 42.5 | 36.7 | |
| AgentlessModel=o3-mini2026.02 | 34.7 | 60.3 | 39.4 | 33.2 | |
| LocAgentModel=GPT-4.12026.02 | 32.3 | 65.6 | 36.7 | 31.2 | |
| LocAgentModel=o3-mini2026.02 | 32.1 | 40.5 | 33.9 | 28.9 | |
| AgentlessModel=Claude-4.5-Sonnet2026.02 | 31.7 | 34.6 | 32 | 27.1 | |
| AgentlessModel=GPT-4o2026.02 | 31.4 | 58.8 | 34.7 | 29.3 | |
| AgentlessModel=GPT-4.12026.02 | 29.3 | 49 | 32.7 | 26.4 |