Code Generation on HumanEval (Speed, Latency, and Resource Usage)
89.02AccuracyMetaGPT
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| MetaGPTImplementation Framework=MASFactory2026.03 | 89.02 | — | — | — | |
| Evo 8BPost-training=SFT, Number of shots=02026.02 | 86.7 | — | — | — | |
| AgentVerseImplementation Framework=original2026.03 | 85 | — | — | — | |
| AgentVerseImplementation Framework=MASFactory2026.03 | 85 | — | — | — | |
| Qwen2.5 7BPost-training=SFT+RL, Number of shots=02026.02 | 84.8 | — | — | — | |
| Vibe Graphing-Task SpecificImplementation Framework=Vibe Graphing2026.03 | 84.76 | — | — | — | |
| Vibe Graphing-ChatDevBase Multi-Agent System=ChatDev, Implementation Framework=Vibe Graphing2026.03 | 83.5 | — | — | — | |
| BD3-LM 7BPost-training=SFT+RL, Number of shots=02026.02 | 82.9 | — | — | — | |
| ChatDevImplementation Framework=original2026.03 | 82.5 | — | — | — | |
| HuggingGPTImplementation Framework=original2026.03 | 82.32 | — | — | — | |
| ChatDevImplementation Framework=MASFactory2026.03 | 81.3 | — | — | — | |
| HuggingGPTImplementation Framework=MASFactory2026.03 | 80.49 | — | — | — | |
| CAMELImplementation Framework=MASFactory2026.03 | 71.85 | — | — | — | |
| MDLM 9BPost-training=SFT+RL, Number of shots=02026.02 | 68.9 | — | — | — | |
| MetaGPTImplementation Framework=original2026.03 | 67.07 | — | — | — | |
| d2CacheFew-shot setting (n-shot)=0-shot, Generation Length=1024, Backbone=Dream-Inst2025.09 | 64.02 | 11.74 | 87.64 | 19.14 | |
| CAMELImplementation Framework=original2026.03 | 62.2 | — | — | — | |
| dLLM-CacheFew-shot setting (n-shot)=0-shot, Generation Length=1024, Backbone=Dream-Inst2025.09 | 60.97 | 3.03 | 338.21 | 19.79 | |
| LLaMA3 8BPost-training=SFT+RL, Number of shots=02026.02 | 59.8 | — | — | — | |
| Fast dLLMFew-shot setting (n-shot)=0-shot, Generation Length=1024, Backbone=Dream-Inst2025.09 | 58.53 | 4.77 | 214.5 | 19.16 | |
| VanillaFew-shot setting (n-shot)=0-shot, Generation Length=1024, Backbone=Dream-Inst2025.09 | 56.71 | 2.62 | 393.47 | 19.06 | |
| LLaDA 8BPost-training=SFT, Number of shots=02026.02 | 47.6 | — | — | — | |
| EGSPO-SAGeneration Length=5122026.03 | 44.5 | — | — | — | |
| EGSPO-SAGeneration Length=Best2026.03 | 44.5 | — | — | — | |
| EGSPO-SAGeneration Length=2562026.03 | 41.5 | — | — | — | |
| EGSPOGeneration Length=2562026.03 | 40.2 | — | — | — | |
| EGSPOGeneration Length=Best2026.03 | 40.2 | — | — | — | |
| EGSPOGeneration Length=5122026.03 | 39.6 | — | — | — | |
| LLaDA-8B-InstructGeneration Length=5122026.03 | 37.8 | — | — | — | |
| LLaDA-8B-InstructGeneration Length=Best2026.03 | 37.8 | — | — | — | |
| d1Generation Length=5122026.03 | 37.8 | — | — | — | |
| d1Generation Length=Best2026.03 | 37.8 | — | — | — | |
| LLaDA-8B-InstructGeneration Length=2562026.03 | 35.3 | — | — | — | |
| d1Generation Length=2562026.03 | 32.9 | — | — | — | |
| EGSPOGeneration Length=1282026.03 | 32.3 | — | — | — | |
| EGSPO-SAGeneration Length=1282026.03 | 32.3 | — | — | — | |
| d1Generation Length=1282026.03 | 31.1 | — | — | — | |
| LLaDA-8B-InstructGeneration Length=1282026.03 | 27.4 | — | — | — | |
| ARD 7BPost-training=SFT+RL, Number of shots=02026.02 | 16.5 | — | — | — |