Software Engineering Question Answering on SWE-QA Pro
51.4Rubric Judge AccuracyPredicate-Based Belief State
Evaluation results
| Method | Links | |
|---|---|---|
| Predicate-Based Belief StateHarness=Iter-RetGen, Representation=Freeform (b_free)2026.05 | 51.4 | |
| BaselineHarness=Iter-RetGen2026.05 | 50.9 | |
| BaselineHarness=MemGPT2026.05 | 48.9 | |
| Predicate-Based Belief StateHarness=Iter-RetGen, Representation=Structured JSON (b_struct)2026.05 | 48.4 | |
| Predicate-Based Belief StateHarness=MemGPT, Representation=Freeform (b_free)2026.05 | 47.7 | |
| LobotomizedHarness=MemGPT2026.05 | 43.6 | |
| Predicate-Based Belief StateHarness=ReAct, Representation=Freeform (b_free)2026.05 | 43 | |
| Predicate-Based Belief StateHarness=MemGPT, Representation=Structured JSON (b_struct)2026.05 | 42.3 | |
| BaselineHarness=ReAct2026.05 | 39.8 | |
| Predicate-Based Belief StateHarness=ReAct, Representation=Structured JSON (b_struct)2026.05 | 37.3 | |
| Predicate-Based Belief StateHarness=IRCoT, Representation=Freeform (b_free)2026.05 | 34.6 | |
| LobotomizedHarness=ReAct2026.05 | 33.7 | |
| Predicate-Based Belief StateHarness=IRCoT, Representation=Structured JSON (b_struct)2026.05 | 33.3 | |
| LobotomizedHarness=IRCoT2026.05 | 31.7 | |
| BaselineHarness=IRCoT2026.05 | 27.3 |