Artifact Finding on QGP-RepoScan
77.8Success RateLangGraph (STATEQGP)
Evaluation Results
| Method | Links | ||||
|---|---|---|---|---|---|
| LangGraph (STATEQGP)Model=gpt-5.4, Task inst.=362026.05 | 77.8 | 41.75 | 0 | 2.537 | |
| Native (STATEQGP)Model=gpt-4.1-mini, Task inst.=362026.05 | 72.2 | 38.67 | 0 | 2.229 | |
| LangGraph (STATEQGP)Model=gpt-4.1-mini, Task inst.=362026.05 | 72.2 | 37.94 | 0 | 2.127 | |
| Native (STATEQGP)Model=gpt-4.1, Task inst.=362026.05 | 72.2 | 38.53 | 0 | 2.276 | |
| LangGraph (STATEQGP)Model=gpt-4.1, Task inst.=362026.05 | 72.2 | 38.92 | 0 | 2.328 | |
| LETTA/MEMGPT (Standard)Model=gpt-5.4, Task inst.=362026.05 | 72.2 | 37.25 | 0.9 | 1.718 | |
| LG+Memory (Standard)Model=gpt-5.4, Task inst.=362026.05 | 72.2 | 36.69 | 0.5 | 3.353 | |
| Native (STATEQGP)Model=gpt-5.4, Task inst.=362026.05 | 69.4 | 37 | 0 | 2.337 | |
| LG+Memory (Standard)Model=gpt-4.1, Task inst.=362026.05 | 47.2 | 31.86 | 0.6 | 3.117 | |
| Native (Verifier-gated)Model=gpt-5.4, Task inst.=362026.05 | 47.2 | 26.47 | 21.6 | 0.804 | |
| LETTA/MEMGPT (Standard)Model=gpt-4.1, Task inst.=362026.05 | 44.4 | 31.83 | 3.3 | 0.412 | |
| LangGraph (Verifier-gated)Model=gpt-4.1, Task inst.=362026.05 | 38.9 | 19.33 | 74.7 | 0.589 | |
| Native (Verifier-gated)Model=gpt-4.1, Task inst.=362026.05 | 33.3 | 18.64 | 64 | 0.692 | |
| Native (Standard)Model=gpt-4.1, Task inst.=362026.05 | 30.6 | 17.11 | 55.7 | 0.615 | |
| LangGraph (Standard)Model=gpt-4.1, Task inst.=362026.05 | 30.6 | 17.78 | 68.1 | 0.7 | |
| Native (Standard)Model=gpt-5.4, Task inst.=362026.05 | 30.6 | 20.89 | 11.2 | 1.177 | |
| LangGraph (Verifier-gated)Model=gpt-5.4, Task inst.=362026.05 | 22.2 | 14.81 | 74.3 | 0.374 | |
| LangGraph (Standard)Model=gpt-5.4, Task inst.=362026.05 | 16.7 | 11.83 | 66.3 | 0.48 | |
| Native (Verifier-gated)Model=gpt-4.1-mini, Task inst.=362026.05 | 8.3 | 6.64 | 87.2 | 0.128 | |
| LangGraph (Standard)Model=gpt-4.1-mini, Task inst.=362026.05 | 8.3 | 5.36 | 86.8 | 0.106 | |
| LG+Memory (Standard)Model=gpt-4.1-mini, Task inst.=362026.05 | 5.6 | 2.25 | 46.8 | 0.389 | |
| Native (Standard)Model=gpt-4.1-mini, Task inst.=362026.05 | 2.8 | 6.42 | 88 | 0.103 | |
| LangGraph (Verifier-gated)Model=gpt-4.1-mini, Task inst.=362026.05 | 2.8 | 6.44 | 88.1 | 0.103 | |
| LETTA/MEMGPT (Standard)Model=gpt-4.1-mini, Task inst.=362026.05 | 0 | 2.39 | 12.9 | 0.179 |