Code Generation on BigCodeBench
83.84AccuracyLatentMem
Evaluation results
| Method | Links | |
|---|---|---|
| LatentMemMAS Framework=AutoGen, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-in2026.02 | 83.84 | |
| LatentMemMAS Framework=MacNet, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-in2026.02 | 83.84 | |
| LatentMemMAS Framework=DyLAN, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-out2026.02 | 83.25 | |
| LatentMemMAS Framework=CAMEL, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-out2026.02 | 82.75 | |
| G-MemoryMAS Framework=AutoGen, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-in2026.02 | 82.67 | |
| OAgentMAS Framework=MacNet, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-in2026.02 | 82.28 | |
| G-MemoryMAS Framework=DyLAN, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-out2026.02 | 82.28 | |
| GenerativeMAS Framework=DyLAN, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-out2026.02 | 82.19 | |
| MetaGPTMAS Framework=AutoGen, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-in2026.02 | 82.11 | |
| GenerativeMAS Framework=AutoGen, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-in2026.02 | 82.11 | |
| OAgentMAS Framework=AutoGen, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-in2026.02 | 82.11 | |
| MetaGPTMAS Framework=CAMEL, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-out2026.02 | 81.93 | |
| G-MemoryMAS Framework=CAMEL, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-out2026.02 | 81.93 | |
| VoyagerMAS Framework=CAMEL, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-out2026.02 | 81.75 | |
| VoyagerMAS Framework=MacNet, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-in2026.02 | 81.67 | |
| VoyagerMAS Framework=DyLAN, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-out2026.02 | 81.58 | |
| VoyagerMAS Framework=AutoGen, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-in2026.02 | 81.49 | |
| OAgentMAS Framework=CAMEL, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-out2026.02 | 81.49 | |
| G-MemoryMAS Framework=MacNet, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-in2026.02 | 81.46 | |
| OAgentMAS Framework=DyLAN, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-out2026.02 | 81.3 | |
| MetaGPTMAS Framework=MacNet, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-in2026.02 | 81.28 | |
| MetaGPTMAS Framework=DyLAN, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-out2026.02 | 81.28 | |
| GenerativeMAS Framework=MacNet, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-in2026.02 | 81.11 | |
| No-memoryMAS Framework=AutoGen, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-in2026.02 | 80.96 | |
| No-memoryMAS Framework=CAMEL, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-out2026.02 | 80.28 | |
| GenerativeMAS Framework=CAMEL, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-out2026.02 | 80.28 | |
| No-memoryMAS Framework=DyLAN, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-out2026.02 | 80.26 | |
| No-memoryMAS Framework=MacNet, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-in2026.02 | 79.28 | |
| CAMELImplementation Framework=MASFactory2026.03 | 78.16 | |
| MiMo-V2-Flash Base# Shots=0-shot, # Activated Params=15B, # Total Params=309B2026.01 | 70.1 | |
| AgentVerseImplementation Framework=original2026.03 | 65.92 | |
| AgentVerseImplementation Framework=MASFactory2026.03 | 64.12 | |
| CAMELImplementation Framework=original2026.03 | 63.51 | |
| DeepSeek-V3.1 Base# Shots=0-shot, # Activated Params=37B, # Total Params=671B2026.01 | 63 | |
| DeepSeek-V3.2 Exp Base# Shots=0-shot, # Activated Params=37B, # Total Params=671B2026.01 | 62.9 | |
| Kimi-K2 Base# Shots=0-shot, # Activated Params=32B, # Total Params=1043B2026.01 | 61.7 | |
| ChatDevImplementation Framework=MASFactory2026.03 | 53.3 | |
| MetaGPTImplementation Framework=MASFactory2026.03 | 51.7 | |
| Vibe Graphing-Task SpecificImplementation Framework=Vibe Graphing2026.03 | 51.67 | |
| ChatDevImplementation Framework=original2026.03 | 50.7 | |
| MetaGPTImplementation Framework=original2026.03 | 50.1 | |
| Vibe Graphing-ChatDevBase Multi-Agent System=ChatDev, Implementation Framework=Vibe Graphing2026.03 | 45.3 | |
| DataFlow-Code-10KBase Model=Qwen2.5-14B-Instruct2025.12 | 41.9 | |
| DataFlow-Code-1KBase Model=Qwen2.5-14B-Instruct2025.12 | 41.4 | |
| DataFlow-Code-5KBase Model=Qwen2.5-14B-Instruct2025.12 | 41.1 | |
| Conductorparameters=7B2025.12 | 37.86 | |
| Gemini 2.5 Pro2025.12 | 37.51 | |
| Qwen2.5-14B-InstructBase Model=Qwen2.5-14B-Instruct, SFT Dataset=Original2025.12 | 37.5 | |
| Claude Sonnet 42025.12 | 37.16 | |
| Code Alpaca-1KBase Model=Qwen2.5-14B-Instruct2025.12 | 37 | |
| Self-OSSBase Model=Qwen2.5-14B-Instruct2025.12 | 36.9 | |
| DataFlow-Code-10KBase Model=Qwen2.5-7B-Instruct2025.12 | 36.8 | |
| DataFlow-Code-5KBase Model=Qwen2.5-7B-Instruct2025.12 | 36.2 | |
| DAJEvaluation Protocol=Test-time Scaling2026.01 | 35.9 | |
| DataFlow-Code-1KBase Model=Qwen2.5-7B-Instruct2025.12 | 35.5 | |
| Qwen2.5-7B-InstructBase Model=Qwen2.5-7B-Instruct, SFT Dataset=Original2025.12 | 35.3 | |
| Skywork-o1 PRMEvaluation Protocol=Test-time Scaling2026.01 | 35.2 | |
| SWIFTEvaluation Protocol=Out-of-distribution2026.04 | 34.3 | |
| DeepSeek GRMEvaluation Protocol=Test-time Scaling2026.01 | 33.8 | |
| Code Alpaca-1KBase Model=Qwen2.5-7B-Instruct2025.12 | 33.3 | |
| o3-mini (medium)Evaluation Protocol=Zero-shot2026.01 | 33.1 | |
| R1-Distill-Qwen-32B2025.12 | 33.07 | |
| GPT 52025.12 | 32.75 | |
| o1 (high)Evaluation Protocol=Zero-shot2026.01 | 32.4 | |
| o3-mini (high)Evaluation Protocol=Zero-shot2026.01 | 32.4 | |
| Claude-3.7-SonnetEvaluation Protocol=Zero-shot2026.01 | 32.4 | |
| Self-OSSBase Model=Qwen2.5-7B-Instruct2025.12 | 31.9 | |
| VanillaEvaluation Protocol=Out-of-distribution2026.04 | 31.6 | |
| Qwen3-32B2025.12 | 30.41 | |
| HuggingGPTImplementation Framework=MASFactory2026.03 | 29.91 | |
| HuggingGPTImplementation Framework=original2026.03 | 28.42 | |
| Qwen3-32B (thinking)mode=thinking2025.12 | 28.38 | |
| CLEAR (Lambert)Budget Strategy=best-of-4 completion-token budgets2026.06 | 19.8 | |
| UniformBudget Strategy=best-of-4 completion-token budgets2026.06 | 16.2 | |
| gemma-3-27b-itinstruct=true2025.12 | 14.86 |