Function-level Code Localization on SWE-bench Live Lite
74.8Acc@1Repo-Enc
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| Repo-EncModel=Claude-4.5-Sonnet2026.02 | 74.8 | 90.4 | 80.7 | 63.3 | |
| Repo-EncModel=GPT-52026.02 | 71.9 | 87.8 | 78.1 | 61.1 | |
| OrcaLocaModel=Claude-4.5-Sonnet2026.02 | 65.1 | 69.4 | 67.8 | 46.1 | |
| Repo-EncModel=GPT-4.12026.02 | 64.7 | 81.9 | 72.1 | 52.6 | |
| OrcaLocaModel=GPT-52026.02 | 59.6 | 74 | 68.6 | 46.6 | |
| Repo-EncModel=o3-mini2026.02 | 56.5 | 75.6 | 64.7 | 46.9 | |
| CoSILModel=GPT-52026.02 | 55.2 | 76.2 | 62.3 | 46.5 | |
| CoSILModel=Claude-4.5-Sonnet2026.02 | 51.1 | 74.9 | 60.1 | 39.6 | |
| Repo-EncModel=GPT-4o2026.02 | 50.5 | 69.4 | 59.4 | 41.8 | |
| CoSILModel=GPT-4.12026.02 | 48.8 | 72.2 | 58.3 | 41.2 | |
| LocAgentModel=GPT-52026.02 | 48 | 68.7 | 56.6 | 40.5 | |
| LocAgentModel=Claude-4.5-Sonnet2026.02 | 47.3 | 60.5 | 52.6 | 39.3 | |
| CoSILModel=o3-mini2026.02 | 43.8 | 65.1 | 51.4 | 35.6 | |
| LocAgentModel=GPT-4.12026.02 | 43.4 | 68.7 | 52.5 | 38.7 | |
| OrcaLocaModel=GPT-4.12026.02 | 42 | 50.5 | 46.2 | 29.1 | |
| AgentlessModel=Claude-4.5-Sonnet2026.02 | 41.4 | 72.4 | 55.3 | 35.9 | |
| CoSILModel=GPT-4o2026.02 | 41.2 | 61.6 | 49.1 | 29.4 | |
| AgentlessModel=GPT-52026.02 | 38.8 | 64.6 | 49.7 | 31.6 | |
| LocAgentModel=GPT-4o2026.02 | 35.7 | 56.4 | 44.5 | 29.9 | |
| AgentlessModel=GPT-4.12026.02 | 35.1 | 59.4 | 46 | 25.4 | |
| AgentlessModel=GPT-4o2026.02 | 30.6 | 57.4 | 41.4 | 26.4 | |
| AgentlessModel=o3-mini2026.02 | 28.8 | 54.2 | 39.3 | 25.6 | |
| OrcaLocaModel=GPT-4o2026.02 | 28.2 | 37 | 32.5 | 21.1 | |
| LocAgentModel=o3-mini2026.02 | 23.8 | 31 | 26.6 | 17.7 | |
| OrcaLocaModel=o3-mini2026.02 | 23.1 | 26.1 | 25.3 | 15.6 |