Code Generation on KodCode
80.96AccuracyReMe (dynamic)
Evaluation results
| Method | Links | ||
|---|---|---|---|
| ReMe (dynamic)MAS=CAMEL, Atom.=✓, Coord.=–, T-free=✓, Backbone=Qwen/Qwen3-4B-Instruct-25072026.06 | 80.96 | — | |
| ConMemMAS=AutoGen, Atom.=✓, Coord.=✓, T-free=✓, Backbone=Qwen/Qwen3-4B-Instruct-25072026.06 | 80.8 | — | |
| ConMemMAS=CAMEL, Atom.=✓, Coord.=✓, T-free=✓, Backbone=Qwen/Qwen3-4B-Instruct-25072026.06 | 80.45 | — | |
| ConMemMAS=MacNet, Atom.=✓, Coord.=✓, T-free=✓, Backbone=Qwen/Qwen3-4B-Instruct-25072026.06 | 79.65 | — | |
| ReMe (dynamic)MAS=AutoGen, Atom.=✓, Coord.=–, T-free=✓, Backbone=Qwen/Qwen3-4B-Instruct-25072026.06 | 79.32 | — | |
| LatentMemMAS=MacNet, Atom.=–, Coord.=✓, T-free=–, Backbone=Qwen/Qwen3-4B-Instruct-25072026.06 | 78.9 | — | |
| ReMe (dynamic)MAS=MacNet, Atom.=✓, Coord.=–, T-free=✓, Backbone=Qwen/Qwen3-4B-Instruct-25072026.06 | 77.9 | — | |
| LatentMemMAS=CAMEL, Atom.=–, Coord.=✓, T-free=–, Backbone=Qwen/Qwen3-4B-Instruct-25072026.06 | 77.75 | — | |
| LatentMemMAS=AutoGen, Atom.=–, Coord.=✓, T-free=–, Backbone=Qwen/Qwen3-4B-Instruct-25072026.06 | 76.8 | — | |
| G-MemoryMAS=MacNet, Atom.=✓, Coord.=–, T-free=✓, Backbone=Qwen/Qwen3-4B-Instruct-25072026.06 | 72.5 | — | |
| MetaGPTMAS=MacNet, Atom.=–, Coord.=–, T-free=✓, Backbone=Qwen/Qwen3-4B-Instruct-25072026.06 | 71.5 | — | |
| G-MemoryMAS=AutoGen, Atom.=✓, Coord.=–, T-free=✓, Backbone=Qwen/Qwen3-4B-Instruct-25072026.06 | 71.4 | — | |
| OAgentsMAS=CAMEL, Atom.=✓, Coord.=–, T-free=✓, Backbone=Qwen/Qwen3-4B-Instruct-25072026.06 | 71.4 | — | |
| OAgentsMAS=MacNet, Atom.=✓, Coord.=–, T-free=✓, Backbone=Qwen/Qwen3-4B-Instruct-25072026.06 | 71.3 | — | |
| GenerativeMAS=MacNet, Atom.=✓, Coord.=–, T-free=✓, Backbone=Qwen/Qwen3-4B-Instruct-25072026.06 | 71.15 | — | |
| GenerativeMAS=AutoGen, Atom.=✓, Coord.=–, T-free=✓, Backbone=Qwen/Qwen3-4B-Instruct-25072026.06 | 70.9 | — | |
| JoyAgentMAS=AutoGen, Atom.=✓, Coord.=–, T-free=✓, Backbone=Qwen/Qwen3-4B-Instruct-25072026.06 | 70.9 | — | |
| OAgentsMAS=AutoGen, Atom.=✓, Coord.=–, T-free=✓, Backbone=Qwen/Qwen3-4B-Instruct-25072026.06 | 70.8 | — | |
| JoyAgentMAS=MacNet, Atom.=✓, Coord.=–, T-free=✓, Backbone=Qwen/Qwen3-4B-Instruct-25072026.06 | 70.8 | — | |
| No-memoryMAS=CAMEL, Atom.=–, Coord.=–, T-free=–, Backbone=Qwen/Qwen3-4B-Instruct-25072026.06 | 70.7 | — | |
| GenerativeMAS=CAMEL, Atom.=✓, Coord.=–, T-free=✓, Backbone=Qwen/Qwen3-4B-Instruct-25072026.06 | 70.65 | — | |
| ChatDevMAS=MacNet, Atom.=–, Coord.=–, T-free=✓, Backbone=Qwen/Qwen3-4B-Instruct-25072026.06 | 70.5 | — | |
| G-MemoryMAS=CAMEL, Atom.=✓, Coord.=–, T-free=✓, Backbone=Qwen/Qwen3-4B-Instruct-25072026.06 | 70.4 | — | |
| No-memoryMAS=MacNet, Atom.=–, Coord.=–, T-free=–, Backbone=Qwen/Qwen3-4B-Instruct-25072026.06 | 70.4 | — | |
| JoyAgentMAS=CAMEL, Atom.=✓, Coord.=–, T-free=✓, Backbone=Qwen/Qwen3-4B-Instruct-25072026.06 | 70.2 | — | |
| MetaGPTMAS=AutoGen, Atom.=–, Coord.=–, T-free=✓, Backbone=Qwen/Qwen3-4B-Instruct-25072026.06 | 70.05 | — | |
| MetaGPTMAS=CAMEL, Atom.=–, Coord.=–, T-free=✓, Backbone=Qwen/Qwen3-4B-Instruct-25072026.06 | 69.9 | — | |
| VoyagerMAS=CAMEL, Atom.=✓, Coord.=–, T-free=✓, Backbone=Qwen/Qwen3-4B-Instruct-25072026.06 | 69.85 | — | |
| VoyagerMAS=MacNet, Atom.=✓, Coord.=–, T-free=✓, Backbone=Qwen/Qwen3-4B-Instruct-25072026.06 | 69.8 | — | |
| VoyagerMAS=AutoGen, Atom.=✓, Coord.=–, T-free=✓, Backbone=Qwen/Qwen3-4B-Instruct-25072026.06 | 68.95 | — | |
| ChatDevMAS=AutoGen, Atom.=–, Coord.=–, T-free=✓, Backbone=Qwen/Qwen3-4B-Instruct-25072026.06 | 68.55 | — | |
| No-memoryMAS=AutoGen, Atom.=–, Coord.=–, T-free=–, Backbone=Qwen/Qwen3-4B-Instruct-25072026.06 | 68.4 | — | |
| ChatDevMAS=CAMEL, Atom.=–, Coord.=–, T-free=✓, Backbone=Qwen/Qwen3-4B-Instruct-25072026.06 | 68.2 | — | |
| LatentMemMAS Framework=AutoGen, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-in2026.02 | 65.9 | — | |
| LatentMemMAS Framework=MacNet, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-in2026.02 | 65.5 | — | |
| LatentMemMAS Framework=DyLAN, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-out2026.02 | 64.25 | — | |
| LatentMemMAS Framework=CAMEL, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-out2026.02 | 63.85 | — | |
| SimpleMemMAS=CAMEL, Atom.=✓, Coord.=–, T-free=✓, Backbone=Qwen/Qwen3-4B-Instruct-25072026.06 | 58.45 | — | |
| MemGenBackbone=Llama 3.1 8B Instruct2026.01 | 57.7 | — | |
| FlashMemBackbone=Llama 3.1 8B Instruct2026.01 | 54.27 | — | |
| SimpleMemMAS=AutoGen, Atom.=✓, Coord.=–, T-free=✓, Backbone=Qwen/Qwen3-4B-Instruct-25072026.06 | 52.3 | — | |
| MetaGPTMAS Framework=CAMEL, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-out2026.02 | 51.7 | — | |
| MetaGPTMAS Framework=MacNet, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-in2026.02 | 50.3 | — | |
| G-MemoryMAS Framework=AutoGen, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-in2026.02 | 50.2 | — | |
| MetaGPTMAS Framework=AutoGen, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-in2026.02 | 50 | — | |
| OAgentMAS Framework=CAMEL, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-out2026.02 | 49.9 | — | |
| GenerativeMAS Framework=MacNet, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-in2026.02 | 49.85 | — | |
| MetaGPTMAS Framework=DyLAN, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-out2026.02 | 49.85 | — | |
| GenerativeMAS Framework=CAMEL, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-out2026.02 | 49.8 | — | |
| OAgentMAS Framework=AutoGen, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-in2026.02 | 49.55 | — | |
| G-MemoryMAS Framework=CAMEL, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-out2026.02 | 49.5 | — | |
| GenerativeMAS Framework=AutoGen, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-in2026.02 | 49.4 | — | |
| G-MemoryMAS Framework=MacNet, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-in2026.02 | 49.2 | — | |
| No-memoryMAS Framework=CAMEL, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-out2026.02 | 48.95 | — | |
| VoyagerMAS Framework=CAMEL, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-out2026.02 | 48.95 | — | |
| G-MemoryMAS Framework=DyLAN, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-out2026.02 | 48.8 | — | |
| No-memoryMAS Framework=MacNet, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-in2026.02 | 48.5 | — | |
| OAgentMAS Framework=MacNet, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-in2026.02 | 48.5 | — | |
| OAgentMAS Framework=DyLAN, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-out2026.02 | 48.25 | — | |
| No-memoryMAS Framework=DyLAN, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-out2026.02 | 47.55 | — | |
| No-memoryMAS Framework=AutoGen, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-in2026.02 | 47.45 | — | |
| GenerativeMAS Framework=DyLAN, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-out2026.02 | 47.1 | — | |
| VoyagerMAS Framework=AutoGen, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-in2026.02 | 46.45 | — | |
| SimpleMemMAS=MacNet, Atom.=✓, Coord.=–, T-free=✓, Backbone=Qwen/Qwen3-4B-Instruct-25072026.06 | 46.1 | — | |
| VoyagerMAS Framework=MacNet, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-in2026.02 | 44.8 | — | |
| VoyagerMAS Framework=DyLAN, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-out2026.02 | 44.8 | — | |
| SnapKVBackbone=Llama 3.1 8B Instruct2026.01 | 40.88 | — | |
| CoT-SCBackbone=Llama 3.1 8B Instruct2026.01 | 37.9 | — | |
| VanillaBackbone=Llama 3.1 8B Instruct2026.01 | 34.61 | — | |
| FlashMemBackbone=Llama 3.2 3B Instruct2026.01 | 29.98 | — | |
| MemGenBackbone=Llama 3.2 3B Instruct2026.01 | 29.64 | — | |
| StableDRLGeneration Length=2562026.05 | 28.2 | — | |
| GDPOGeneration Length=5122026.05 | 28 | — | |
| StableDRLGeneration Length=5122026.05 | 27.6 | — | |
| wd1Generation Length=5122026.05 | 27.4 | — | |
| SPGGeneration Length=5122026.05 | 27 | — | |
| GDPOGeneration Length=2562026.05 | 26.8 | — | |
| d1Generation Length=5122026.05 | 26.6 | — | |
| Diffu-GRPOGeneration Length=5122026.05 | 26.4 | — | |
| ESPOGeneration Length=5122026.05 | 26.4 | — | |
| SPGGeneration Length=2562026.05 | 26.4 | — | |
| CoT-SCBackbone=Llama 3.2 3B Instruct2026.01 | 26.27 | — | |
| LLaDA-8B-InstructGeneration Length=5122026.05 | 26.2 | — | |
| SFTGeneration Length=5122026.05 | 25.8 | — | |
| ESPOGeneration Length=2562026.05 | 25.8 | — | |
| wd1Generation Length=2562026.05 | 25.6 | — | |
| d1Generation Length=2562026.05 | 25.2 | — | |
| AdaBlock-dLLMGeneration Length=5122026.05 | 24.8 | — | |
| Diffu-GRPOGeneration Length=2562026.05 | 24.6 | — | |
| AdaBlock-dLLMGeneration Length=2562026.05 | 22.4 | — | |
| LLaDA-8B-InstructGeneration Length=2562026.05 | 22 | — | |
| VanillaBackbone=Llama 3.2 3B Instruct2026.01 | 21.93 | — | |
| SnapKVBackbone=Llama 3.2 3B Instruct2026.01 | 20.33 | — | |
| SFTGeneration Length=2562026.05 | 19.4 | — | |
| Block-R1Source Domain=Multi-domain, Backbone=LLaDA-8B-Instruct2026.05 | — | 28.6 | |
| StableDRLSource Domain=KodCode, Backbone=LLaDA-8B-Instruct, Post-training Method=RL (StableDRL)2026.05 | — | 28.2 |