Code Generation on MBPP (Pass@1 Python, Pass@1 Rust)
96.5Pass@1 Accuracy (Python)SYMPHONY-L
Evaluation results
| Method | Links | ||
|---|---|---|---|
| SYMPHONY-LSize=Large2026.01 | 96.5 | 97.4 | |
| SYMPHONY-SSize=Small2026.01 | 92.7 | 94.6 | |
| AgentCoder2026.01 | 91.8 | — | |
| MASTER2026.01 | 91 | — | |
| AgentVerse2026.01 | 89 | — | |
| MetaGPT2026.01 | 87.7 | — | |
| LATS2026.01 | 81.1 | — | |
| GPT-42026.01 | 80 | 71 | |
| Reflexion2026.01 | 77.1 | 75.4 | |
| RAP2026.01 | 71.4 | — | |
| GPT-4Prompting=ReAct2026.01 | 71 | — | |
| GPT-4Prompting=CoT2026.01 | 68.3 | — |