Code Generation on HumanEval (Accuracy and Average Cost)
75.76AccuracyMACA (Ours)
Evaluation results
| Method | Links | ||
|---|---|---|---|
| MACA (Ours)Model=Llama-3.1-8B, Structure Adaptation=✓, Orchestration Adaptation=✓2026.05 | 75.76 | 2,100.1 | |
| DyLANModel=Llama-3.1-8B, Structure Adaptation=✗, Orchestration Adaptation=✗2026.05 | 72.73 | 15,256.7 | |
| PuppeteerModel=Llama-3.1-8B, Structure Adaptation=✗, Orchestration Adaptation=✓2026.05 | 71.49 | 2,798.3 | |
| MacNetModel=Llama-3.1-8B, Structure Adaptation=✗, Orchestration Adaptation=✗2026.05 | 68.18 | 8,782.6 | |
| AgentVerseModel=Llama-3.1-8B, Structure Adaptation=✗, Orchestration Adaptation=✗2026.05 | 66.18 | 6,826.9 | |
| MaASModel=Llama-3.1-8B, Structure Adaptation=✓, Orchestration Adaptation=✗2026.05 | 65.15 | 4,796.9 | |
| AgentPruneModel=Llama-3.1-8B, Structure Adaptation=✓, Orchestration Adaptation=✗2026.05 | 62.5 | 2,357 | |
| Self-RefineModel=Llama-3.1-8B, Structure Adaptation=✗, Orchestration Adaptation=✗2026.05 | 62.12 | 1,609.8 | |
| VanillaModel=Llama-3.1-8B, Structure Adaptation=✗, Orchestration Adaptation=✗2026.05 | 60.61 | 256.6 | |
| ComplexCoTModel=Llama-3.1-8B, Structure Adaptation=✗, Orchestration Adaptation=✗2026.05 | 57.58 | 1,154.4 | |
| CoTModel=Llama-3.1-8B, Structure Adaptation=✗, Orchestration Adaptation=✗2026.05 | 56.44 | 297.9 | |
| SC (CoT×5)Model=Llama-3.1-8B, Structure Adaptation=✗, Orchestration Adaptation=✗2026.05 | 55.38 | 2,389.6 |