Module-level Localization on SWE-Bench-Lite (test)
77.73NDCG@5LocAgent
Evaluation results
| Method | Links | ||
|---|---|---|---|
| LocAgentType=Agent-Based, Loc-Model=Claude-3.52025.03 | 77.73 | 78.1 | |
| LocAgentType=Agent-Based, Loc-Model=Qwen2.5-32B(ft)2025.03 | 76.28 | 76.77 | |
| OpenhandsType=Agent-Based, Loc-Model=Claude-3.52025.03 | 75.79 | 75.92 | |
| SWE-agentType=Agent-Based, Loc-Model=Claude-3.52025.03 | 72.77 | 72.9 | |
| LocAgentType=Agent-Based, Loc-Model=Qwen2.5-7B(ft)2025.03 | 70.99 | 71.68 | |
| Moatless ToolsType=Agent-Based, Loc-Model=Claude-3.52025.03 | 69.11 | 69.11 | |
| Moatless ToolsType=Agent-Based, Loc-Model=GPT-4o2025.03 | 68.57 | 69.09 | |
| AgentlessType=Procedure-Based, Loc-Model=Claude-3.52025.03 | 67.36 | 67.36 | |
| AgentlessType=Procedure-Based, Loc-Model=GPT-4o2025.03 | 64.31 | 64.31 | |
| OpenhandsType=Agent-Based, Loc-Model=GPT-4o2025.03 | 58.18 | 58.6 | |
| CodeRankEmbedType=Embedding-Based2025.03 | 57.51 | 59.76 | |
| SWE-agentType=Agent-Based, Loc-Model=GPT-4o2025.03 | 53.95 | 53.95 | |
| E5-base-v2Type=Embedding-Based2025.03 | 53.15 | 54.45 | |
| Jina-Code-v2Type=Embedding-Based2025.03 | 51.02 | 54.13 | |
| Codesage-large-v2Type=Embedding-Based2025.03 | 49.38 | 52.22 | |
| BM25Type=Embedding-Based2025.03 | 37.31 | 39.86 |