Repository Reconstruction on RepoCraft (test)
2.14StepsLocAgent
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| LocAgentLLM=o3-mini2026.02 | 2.14 | 0.04 | 21.75 | |
| RPG-EncoderLLM=GPT-52026.02 | 6.34 | 0.22 | 4.25 | |
| LocAgentLLM=GPT-52026.02 | 6.48 | 0.49 | 1.64 | |
| RPG-EncoderLLM=o3-mini2026.02 | 6.53 | 0.11 | 7.07 | |
| RPG-EncoderLLM=GPT-4.12026.02 | 6.75 | 0.18 | 4.63 | |
| LocAgentLLM=Claude-4.52026.02 | 7.54 | 0.88 | 0.85 | |
| RPG-EncoderLLM=GPT-4o2026.02 | 8.22 | 0.2 | 3.84 | |
| LocAgentLLM=GPT-4o2026.02 | 8.95 | 0.5 | 1.35 | |
| OrcaLocaLLM=o3-mini2026.02 | 10.13 | 0.1 | 4.86 | |
| OrcaLocaLLM=Claude-4.52026.02 | 11.32 | 0.88 | 0.91 | |
| LocAgentLLM=GPT-4.12026.02 | 11.94 | 0.86 | 0.76 | |
| CoSILLLM=Claude-4.52026.02 | 12.6 | 0.48 | 1.64 | |
| RPG-EncoderLLM=Claude-4.52026.02 | 16.75 | 1.32 | 0.71 | |
| CoSILLLM=o3-mini2026.02 | 18.95 | 0.18 | 4.07 | |
| CoSILLLM=GPT-52026.02 | 19.52 | 0.31 | 2.64 | |
| CoSILLLM=GPT-4.12026.02 | 19.77 | 0.24 | 3.1 | |
| OrcaLocaLLM=GPT-4.12026.02 | 20.22 | 0.46 | 1.48 | |
| CoSILLLM=GPT-4o2026.02 | 20.65 | 0.3 | 2.21 | |
| OrcaLocaLLM=GPT-4o2026.02 | 29.26 | 0.98 | 0.54 | |
| OrcaLocaLLM=GPT-52026.02 | 36.93 | 0.75 | 1.16 |