Code Generation on BigCodeBench (30% held-out tasks)
55.8Success Rate (SR)APEX-EM (A5: Opus judge)
Evaluation results
| Method | Links | ||
|---|---|---|---|
| APEX-EM (A5: Opus judge)Type=All + Opus judge, Memory=Frozen2026.03 | 55.8 | 9.9 | |
| APEX-EM (A5: GPT 4o as base)Type=All, Base Model=GPT-4o, Memory=Frozen2026.03 | 53.5 | 9.9 | |
| APEX-EM (EG2: Full memory)Type=Rich feedback, iter., all, Memory=Frozen2026.03 | 51.5 | 5.6 | |
| APEX-EM (EG1: Entity graph)Type=Rich feedback, iter., EG, Memory=Frozen2026.03 | 51.2 | 5.3 | |
| MemRLType=Q-value episodic memory, Base Model=GPT-4o, Memory=Frozen2026.03 | 50.8 | 2.3 | |
| APEX-EM (A1: Memory, no judge)Type=Memory, binary signal, Memory=Frozen2026.03 | 50.6 | 4.7 | |
| APEX-EM (R1: Semantic only)Type=Rich feedback, iter., semantic, Memory=Frozen2026.03 | 50.3 | 4.4 | |
| Self-RAGType=Critique-filtered retrieval, Base Model=GPT-4o, Memory=Frozen2026.03 | 50 | 1.5 | |
| MemPType=Procedural distillation, Base Model=GPT-4o, Memory=Frozen2026.03 | 49.4 | 0.9 | |
| APEX-EM (A3: Judge + iteration)Type=Memory, binary, iteration, Memory=Frozen2026.03 | 48.8 | 2.9 | |
| No MemoryType=Single-shot LLM, Base Model=GPT-4o, Memory=Frozen2026.03 | 48.5 | — | |
| Mem0Type=Entity-centric memory, Base Model=GPT-4o, Memory=Frozen2026.03 | 48.5 | 0 | |
| RAGType=Semantic retrieval, Base Model=GPT-4o, Memory=Frozen2026.03 | 47.9 | 0.6 | |
| APEX-EM (A2: Memory + judge)Type=Memory, rich feedback, Memory=Frozen2026.03 | 46.5 | 0.6 | |
| APEX-EM (A0: No Memory)Type=Sonnet 4.5, single-shot, Memory=Frozen2026.03 | 45.9 | — |