File-level Code Localization on SWE-bench Live Lite
82.1Acc@1Repo-Enc
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| Repo-EncModel=GPT-52026.02 | 82.1 | 94.4 | 85.4 | 76.2 | |
| Repo-EncModel=Claude-4.5-Sonnet2026.02 | 82 | 93.9 | 85.6 | 75.8 | |
| LocAgentModel=GPT-52026.02 | 79.7 | 93 | 81.4 | 74.2 | |
| Repo-EncModel=GPT-4.12026.02 | 78 | 90.5 | 81.4 | 69 | |
| LocAgentModel=GPT-4.12026.02 | 74.7 | 87.9 | 76.8 | 66.1 | |
| OrcaLocaModel=Claude-4.5-Sonnet2026.02 | 74.7 | 78.3 | 76.2 | 61.5 | |
| OrcaLocaModel=GPT-52026.02 | 74.4 | 82.3 | 77.6 | 63.5 | |
| Repo-EncModel=o3-mini2026.02 | 73.7 | 88.2 | 77.5 | 64.5 | |
| CoSILModel=GPT-52026.02 | 69.8 | 89.3 | 72.9 | 62.2 | |
| Repo-EncModel=GPT-4o2026.02 | 69.2 | 83.5 | 73.2 | 60.3 | |
| AgentlessModel=GPT-52026.02 | 64.5 | 87.4 | 65.1 | 57.4 | |
| CoSILModel=Claude-4.5-Sonnet2026.02 | 64.5 | 88.3 | 69.4 | 57.5 | |
| AgentlessModel=Claude-4.5-Sonnet2026.02 | 63.8 | 89.7 | 66.1 | 58 | |
| LocAgentModel=GPT-4o2026.02 | 62.5 | 80 | 66.8 | 54.2 | |
| CoSILModel=GPT-4.12026.02 | 62.3 | 84.7 | 67.3 | 55.6 | |
| AgentlessModel=GPT-4.12026.02 | 62 | 85.5 | 63 | 54.5 | |
| CoSILModel=o3-mini2026.02 | 60.9 | 80.8 | 66.1 | 54.8 | |
| CoSILModel=GPT-4o2026.02 | 60.1 | 77 | 63.7 | 50.7 | |
| LocAgentModel=Claude-4.5-Sonnet2026.02 | 58.7 | 69 | 61.6 | 54.7 | |
| OrcaLocaModel=GPT-4.12026.02 | 56.2 | 59.6 | 57.1 | 44.2 | |
| AgentlessModel=GPT-4o2026.02 | 56.1 | 78.8 | 57.1 | 48.3 | |
| AgentlessModel=o3-mini2026.02 | 54.2 | 78.5 | 55.6 | 47.7 | |
| LocAgentModel=o3-mini2026.02 | 47.6 | 59.4 | 49.7 | 41.2 | |
| OrcaLocaModel=GPT-4o2026.02 | 42.5 | 47.6 | 45 | 34 | |
| OrcaLocaModel=o3-mini2026.02 | 35.4 | 38 | 36.2 | 27.6 |