Code Generation on BigCodeBench Lite-Pro Compositional Stream
66.7AccuracyMEMPROBE
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| MEMPROBEEvaluation Pass=1st2026.06 | 66.7 | 21.9 | — | |
| MEMPROBEEvaluation Pass=2nd2026.06 | 64.6 | — | 2.1 | |
| ExpRAGEvaluation Pass=1st2026.06 | 62.5 | 17.7 | — | |
| ExpRAGEvaluation Pass=2nd2026.06 | 62.5 | — | 0 | |
| Self-invokingEvaluation Pass=1st, Self-invoking setting=true2026.06 | 60.4 | — | — | |
| Self-invokingEvaluation Pass=2nd, Self-invoking setting=true2026.06 | 60.4 | — | — | |
| Self-invoking w/ subtask solutionEvaluation Pass=1st, Reference subtask context=subtask solution2026.06 | 59 | — | — | |
| Self-invoking w/ subtask solutionEvaluation Pass=2nd, Reference subtask context=subtask solution2026.06 | 59 | — | — | |
| ReMemEvaluation Pass=1st2026.06 | 58.3 | 13.5 | — | |
| ReMemEvaluation Pass=2nd2026.06 | 56.3 | — | 2 | |
| DC-RSEvaluation Pass=2nd2026.06 | 50.7 | — | 2.1 | |
| DC-RSEvaluation Pass=1st2026.06 | 48.6 | 3.8 | — | |
| LangMemEvaluation Pass=2nd2026.06 | 45.8 | — | 1.4 | |
| ReActEvaluation Pass=1st2026.06 | 44.8 | — | — | |
| ReActEvaluation Pass=2nd2026.06 | 44.8 | — | — | |
| LangMemEvaluation Pass=1st2026.06 | 44.4 | 0.4 | — | |
| Mem0Evaluation Pass=2nd2026.06 | 44.4 | — | 0.6 | |
| Mem0Evaluation Pass=1st2026.06 | 43.8 | 1 | — | |
| AWMEvaluation Pass=1st2026.06 | 39.6 | 5.2 | — | |
| AWMEvaluation Pass=2nd2026.06 | 38.2 | — | 1.4 |