Question Answering on PopQA (Score)
52.7ScoreLatentMem
Evaluation results
| Method | Links | |
|---|---|---|
| LatentMemMAS=AutoGen, Atom.=–, Coord.=✓, T-free=–, Backbone=Qwen/Qwen3-4B-Instruct-25072026.06 | 52.7 | |
| ConMemMAS=AutoGen, Atom.=✓, Coord.=✓, T-free=✓, Backbone=Qwen/Qwen3-4B-Instruct-25072026.06 | 49.9 | |
| ReMe (dynamic)MAS=AutoGen, Atom.=✓, Coord.=–, T-free=✓, Backbone=Qwen/Qwen3-4B-Instruct-25072026.06 | 49.1 | |
| LatentMemMAS=CAMEL, Atom.=–, Coord.=✓, T-free=–, Backbone=Qwen/Qwen3-4B-Instruct-25072026.06 | 47.23 | |
| SimpleMemMAS=AutoGen, Atom.=✓, Coord.=–, T-free=✓, Backbone=Qwen/Qwen3-4B-Instruct-25072026.06 | 46.9 | |
| ReMe (dynamic)MAS=MacNet, Atom.=✓, Coord.=–, T-free=✓, Backbone=Qwen/Qwen3-4B-Instruct-25072026.06 | 46.36 | |
| ConMemMAS=CAMEL, Atom.=✓, Coord.=✓, T-free=✓, Backbone=Qwen/Qwen3-4B-Instruct-25072026.06 | 45.73 | |
| ReMe (dynamic)MAS=CAMEL, Atom.=✓, Coord.=–, T-free=✓, Backbone=Qwen/Qwen3-4B-Instruct-25072026.06 | 45.33 | |
| ConMemMAS=MacNet, Atom.=✓, Coord.=✓, T-free=✓, Backbone=Qwen/Qwen3-4B-Instruct-25072026.06 | 45.29 | |
| SimpleMemMAS=MacNet, Atom.=✓, Coord.=–, T-free=✓, Backbone=Qwen/Qwen3-4B-Instruct-25072026.06 | 44.97 | |
| LatentMemMAS=MacNet, Atom.=–, Coord.=✓, T-free=–, Backbone=Qwen/Qwen3-4B-Instruct-25072026.06 | 44.14 | |
| CDKC (w/ 2 round)Backbone=Llama-3.1-8B-Instruct, Phase=Knowledge Calibration2026.02 | 43.93 | |
| G-MemoryMAS=MacNet, Atom.=✓, Coord.=–, T-free=✓, Backbone=Qwen/Qwen3-4B-Instruct-25072026.06 | 43.88 | |
| RAGBackbone=Qwen2.5-7B-Instruct, Phase=Fundamental Capabilities2026.02 | 43.44 | |
| GenerativeMAS=MacNet, Atom.=✓, Coord.=–, T-free=✓, Backbone=Qwen/Qwen3-4B-Instruct-25072026.06 | 43.39 | |
| JoyAgentMAS=MacNet, Atom.=✓, Coord.=–, T-free=✓, Backbone=Qwen/Qwen3-4B-Instruct-25072026.06 | 43.22 | |
| RAGBackbone=Llama-3.1-8B-Instruct, Phase=Fundamental Capabilities2026.02 | 42.73 | |
| G-MemoryMAS=AutoGen, Atom.=✓, Coord.=–, T-free=✓, Backbone=Qwen/Qwen3-4B-Instruct-25072026.06 | 42.67 | |
| SimpleMemMAS=CAMEL, Atom.=✓, Coord.=–, T-free=✓, Backbone=Qwen/Qwen3-4B-Instruct-25072026.06 | 42.26 | |
| OAgentsMAS=MacNet, Atom.=✓, Coord.=–, T-free=✓, Backbone=Qwen/Qwen3-4B-Instruct-25072026.06 | 41.9 | |
| JoyAgentMAS=AutoGen, Atom.=✓, Coord.=–, T-free=✓, Backbone=Qwen/Qwen3-4B-Instruct-25072026.06 | 41.89 | |
| OAgentsMAS=AutoGen, Atom.=✓, Coord.=–, T-free=✓, Backbone=Qwen/Qwen3-4B-Instruct-25072026.06 | 40.7 | |
| GenerativeMAS=AutoGen, Atom.=✓, Coord.=–, T-free=✓, Backbone=Qwen/Qwen3-4B-Instruct-25072026.06 | 40.37 | |
| CDKCBackbone=Llama-3.1-8B-Instruct, Phase=Knowledge Calibration2026.02 | 40.33 | |
| No-memoryMAS=AutoGen, Atom.=–, Coord.=–, T-free=–, Backbone=Qwen/Qwen3-4B-Instruct-25072026.06 | 38.78 | |
| G-MemoryMAS=CAMEL, Atom.=✓, Coord.=–, T-free=✓, Backbone=Qwen/Qwen3-4B-Instruct-25072026.06 | 38.04 | |
| JoyAgentMAS=CAMEL, Atom.=✓, Coord.=–, T-free=✓, Backbone=Qwen/Qwen3-4B-Instruct-25072026.06 | 37.5 | |
| MetaGPTMAS=CAMEL, Atom.=–, Coord.=–, T-free=✓, Backbone=Qwen/Qwen3-4B-Instruct-25072026.06 | 37.47 | |
| GRPOBackbone=Llama-3.1-8B-Instruct, Phase=Knowledge Calibration2026.02 | 37.33 | |
| Olmo 3 Think (DPO)Training Stage=DPO, Model Family=Olmo 3, Parameter Count=32B, Thinking Capability=true2025.12 | 37 | |
| ChatDevMAS=CAMEL, Atom.=–, Coord.=–, T-free=✓, Backbone=Qwen/Qwen3-4B-Instruct-25072026.06 | 36.78 | |
| VoyagerMAS=CAMEL, Atom.=✓, Coord.=–, T-free=✓, Backbone=Qwen/Qwen3-4B-Instruct-25072026.06 | 36.25 | |
| VoyagerMAS=MacNet, Atom.=✓, Coord.=–, T-free=✓, Backbone=Qwen/Qwen3-4B-Instruct-25072026.06 | 35.38 | |
| ChatDevMAS=MacNet, Atom.=–, Coord.=–, T-free=✓, Backbone=Qwen/Qwen3-4B-Instruct-25072026.06 | 35.33 | |
| GenerativeMAS=CAMEL, Atom.=✓, Coord.=–, T-free=✓, Backbone=Qwen/Qwen3-4B-Instruct-25072026.06 | 35.32 | |
| BARRELBackbone=Llama-3.1-8B-Instruct, Phase=Knowledge Calibration2026.02 | 35 | |
| MetaGPTMAS=MacNet, Atom.=–, Coord.=–, T-free=✓, Backbone=Qwen/Qwen3-4B-Instruct-25072026.06 | 33.98 | |
| MetaGPTMAS=AutoGen, Atom.=–, Coord.=–, T-free=✓, Backbone=Qwen/Qwen3-4B-Instruct-25072026.06 | 33.8 | |
| VoyagerMAS=AutoGen, Atom.=✓, Coord.=–, T-free=✓, Backbone=Qwen/Qwen3-4B-Instruct-25072026.06 | 33.56 | |
| ChatDevMAS=AutoGen, Atom.=–, Coord.=–, T-free=✓, Backbone=Qwen/Qwen3-4B-Instruct-25072026.06 | 33.24 | |
| Olmo 3 Think (SFT)Training Stage=SFT, Model Family=Olmo 3, Parameter Count=32B, Thinking Capability=true2025.12 | 33.1 | |
| No-memoryMAS=CAMEL, Atom.=–, Coord.=–, T-free=–, Backbone=Qwen/Qwen3-4B-Instruct-25072026.06 | 32.38 | |
| Qwen 3 VL 32B ThinkModel Family=Qwen 3, Parameter Count=32B, Thinking Capability=true2025.12 | 32.2 | |
| K2-V2 70B InstructModel Family=K2, Parameter Count=70B, Thinking Capability=false2025.12 | 32.2 | |
| OAgentsMAS=CAMEL, Atom.=✓, Coord.=–, T-free=✓, Backbone=Qwen/Qwen3-4B-Instruct-25072026.06 | 31.99 | |
| Olmo 3 Think (Final 3.0)Training Stage=Final Think 3.0, Model Family=Olmo 3, Parameter Count=32B, Thinking Capability=true2025.12 | 31.9 | |
| CDKC (w/ 2 round)Backbone=Qwen2.5-7B-Instruct, Phase=Knowledge Calibration2026.02 | 31.67 | |
| Olmo 3.1 Think 32BTraining Stage=Final Think 3.1, Model Family=Olmo 3.1, Parameter Count=32B, Thinking Capability=true2025.12 | 30.9 | |
| Qwen 3 32BModel Family=Qwen 3, Parameter Count=32B, Thinking Capability=false2025.12 | 30.7 | |
| Vanilla SFTBackbone=Llama-3.1-8B-Instruct, Phase=Knowledge Expansion2026.02 | 29.43 | |
| Qwen 3 VL 8B Think2025.12 | 29.3 | |
| CRew-DPOBackbone=Llama-3.1-8B-Instruct, Phase=Knowledge Calibration2026.02 | 28.97 | |
| Granite 3.3 8B Inststage=Instruct2025.12 | 28.9 | |
| CDKCBackbone=Qwen2.5-7B-Instruct, Phase=Knowledge Calibration2026.02 | 28.27 | |
| GRPOBackbone=Qwen2.5-7B-Instruct, Phase=Knowledge Calibration2026.02 | 28.1 | |
| Know WhatBackbone=Llama-3.1-8B-Instruct, Phase=Knowledge Calibration2026.02 | 26.97 | |
| DS-R1 32BModel Family=DeepSeek-R1, Parameter Count=32B, Thinking Capability=true2025.12 | 26.7 | |
| CGKEBackbone=Llama-3.1-8B-Instruct, Phase=Knowledge Expansion2026.02 | 26.6 | |
| Qwen 3 VL 8B Inststage=Instruct2025.12 | 26.5 | |
| OLMo 2 7B Inststage=Instruct2025.12 | 25.5 | |
| Apertus 8B Inststage=Instruct2025.12 | 25.5 | |
| LLKD-SFTBackbone=Llama-3.1-8B-Instruct, Phase=Knowledge Expansion2026.02 | 25.07 | |
| No-memoryMAS=MacNet, Atom.=–, Coord.=–, T-free=–, Backbone=Qwen/Qwen3-4B-Instruct-25072026.06 | 24.89 | |
| Olmo 3 7B ThinkStage=DPO2025.12 | 24.7 | |
| CoTBackbone=Llama-3.1-8B-Instruct, Phase=Fundamental Capabilities2026.02 | 24.33 | |
| Qwen 3 8B2025.12 | 24.3 | |
| Olmo 3 7B ThinkStage=Final Think2025.12 | 23.7 | |
| Qwen 2.5 7Bstage=Instruct2025.12 | 21.5 | |
| BARRELBackbone=Qwen2.5-7B-Instruct, Phase=Knowledge Calibration2026.02 | 20.97 | |
| Olmo 3 7B ThinkStage=SFT2025.12 | 20.8 | |
| Olmo 3 7B Instructstage=DPO2025.12 | 20.7 | |
| Qwen 3 8Bstage=Instruct2025.12 | 20.4 | |
| Vanilla LLMBackbone=Llama-3.1-8B-Instruct, Phase=Fundamental Capabilities2026.02 | 20.33 | |
| OpenThinker3 7B2025.12 | 18 | |
| Nemotron Nano 9B v22025.12 | 17.9 | |
| CRew-DPOBackbone=Qwen2.5-7B-Instruct, Phase=Knowledge Calibration2026.02 | 16.57 | |
| Know WhatBackbone=Qwen2.5-7B-Instruct, Phase=Knowledge Calibration2026.02 | 16.53 | |
| Olmo 3 7B Instructstage=SFT2025.12 | 16.5 | |
| CGKEBackbone=Qwen2.5-7B-Instruct, Phase=Knowledge Expansion2026.02 | 16.13 | |
| Vanilla SFTBackbone=Qwen2.5-7B-Instruct, Phase=Knowledge Expansion2026.02 | 15.9 | |
| LLKD-SFTBackbone=Qwen2.5-7B-Instruct, Phase=Knowledge Expansion2026.02 | 15.43 | |
| CoTBackbone=Qwen2.5-7B-Instruct, Phase=Fundamental Capabilities2026.02 | 15.07 | |
| OR Nemotron 7B2025.12 | 14.5 | |
| Vanilla LLMBackbone=Qwen2.5-7B-Instruct, Phase=Fundamental Capabilities2026.02 | 14.5 | |
| Olmo 3 7B Instructstage=Final Instruct2025.12 | 14.1 | |
| DS-R1 Qwen 7B2025.12 | 12.8 |