Function-level Localization on SWE-Bench Lite latest (test)
64.34NDCG@5LocAgent
Evaluation results
| Method | Links | ||
|---|---|---|---|
| LocAgentType=Agent-Based, Loc-Model=Claude-3.52025.03 | 64.34 | 65.57 | |
| LocAgentType=Agent-Based, Loc-Model=Qwen2.5-32B(ft)2025.03 | 64.27 | 65.93 | |
| OpenhandsType=Agent-Based, Loc-Model=Claude-3.52025.03 | 63.13 | 63.8 | |
| SWE-agentType=Agent-Based, Loc-Model=Claude-3.52025.03 | 59.67 | 59.79 | |
| AgentlessType=Procedure-Based, Loc-Model=Claude-3.52025.03 | 57.55 | 57.55 | |
| LocAgentType=Agent-Based, Loc-Model=Qwen2.5-7B(ft)2025.03 | 55.62 | 58.09 | |
| AgentlessType=Procedure-Based, Loc-Model=GPT-4o2025.03 | 53.81 | 53.81 | |
| Moatless ToolsType=Agent-Based, Loc-Model=Claude-3.52025.03 | 53.03 | 53.16 | |
| Moatless ToolsType=Agent-Based, Loc-Model=GPT-4o2025.03 | 49.77 | 50.62 | |
| OpenhandsType=Agent-Based, Loc-Model=GPT-4o2025.03 | 44.34 | 44.66 | |
| SWE-agentType=Agent-Based, Loc-Model=GPT-4o2025.03 | 42.32 | 42.44 | |
| CodeRankEmbedType=Embedding-Based2025.03 | 40.28 | 42.55 | |
| Jina-Code-v2Type=Embedding-Based2025.03 | 33.28 | 36.44 | |
| E5-base-v2Type=Embedding-Based2025.03 | 31.39 | 35.3 | |
| Codesage-large-v2Type=Embedding-Based2025.03 | 27.03 | 30.74 | |
| BM25Type=Embedding-Based2025.03 | 26.15 | 27.92 |