Code Generation on HumanEval-Pro (Held-out after Naive Stream)
79.2AccuracySelf-invoking
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Self-invokingSelf-invoking setting=true2026.06 | 79.2 | — | |
| Self-invoking w/ subtask solutionReference subtask context=subtask solution2026.06 | 73.3 | — | |
| ReActcontext=Held-out2026.06 | 72.5 | — | |
| MEMPROBEcontext=Held-out2026.06 | 70.8 | 1.7 | |
| AWMcontext=Held-out2026.06 | 70 | 2.5 | |
| ReMemcontext=Held-out2026.06 | 70 | 2.5 | |
| Mem0context=Held-out2026.06 | 69.2 | 3.3 | |
| ExpRAGcontext=Held-out2026.06 | 69.2 | 3.3 | |
| LangMemcontext=Held-out2026.06 | 66.7 | 5.8 | |
| DC-RScontext=Held-out2026.06 | 65.8 | 6.7 |