File-level Localization on SWE-Bench-Lite latest (test)
77.74NDCG@1LocAgent
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| LocAgentType=Agent-Based, Loc-Model=Claude-3.52025.03 | 77.74 | 86.19 | 87.14 | |
| SWE-agentType=Agent-Based, Loc-Model=Claude-3.52025.03 | 77.37 | 84.32 | 84.93 | |
| OpenhandsType=Agent-Based, Loc-Model=Claude-3.52025.03 | 76.28 | 84.27 | 84.43 | |
| LocAgentType=Agent-Based, Loc-Model=Qwen2.5-32B(ft)2025.03 | 75.91 | 84.74 | 85.64 | |
| Moatless ToolsType=Agent-Based, Loc-Model=GPT-4o2025.03 | 73.36 | 80.03 | 80.33 | |
| AgentlessType=Procedure-Based, Loc-Model=Claude-3.52025.03 | 72.63 | 76.72 | 76.87 | |
| Moatless ToolsType=Agent-Based, Loc-Model=Claude-3.52025.03 | 72.63 | 80.73 | 80.88 | |
| LocAgentType=Agent-Based, Loc-Model=Qwen2.5-7B(ft)2025.03 | 70.8 | 79.36 | 80.9 | |
| AgentlessType=Procedure-Based, Loc-Model=GPT-4o2025.03 | 67.15 | 71.76 | 71.76 | |
| OpenhandsType=Agent-Based, Loc-Model=GPT-4o2025.03 | 60.95 | 67.62 | 68.39 | |
| SWE-agentType=Agent-Based, Loc-Model=GPT-4o2025.03 | 57.3 | 63.96 | 64.12 | |
| CodeRankEmbedType=Embedding-Based2025.03 | 52.55 | 67.54 | 70.39 | |
| E5-base-v2Type=Embedding-Based2025.03 | 49.64 | 64.19 | 66.6 | |
| Codesage-large-v2Type=Embedding-Based2025.03 | 47.81 | 60.82 | 64.39 | |
| Jina-Code-v2Type=Embedding-Based2025.03 | 43.43 | 59.93 | 63.7 | |
| BM25Type=Embedding-Based2025.03 | 38.69 | 46.5 | 50.61 |