Embodied AI Task Planning on EB-ALFRED
82Average ScoreA-Mem
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| A-MemExecutor=Gemini2026.05 | 82 | 90 | 84 | 86 | 78 | 82 | 72 | |
| G-MemExecutor=Gemini2026.05 | 79.27 | 82 | 80 | 78 | 82 | 76 | 78 | |
| LangMemExecutor=Gemini2026.05 | 78 | 82 | 78 | 84 | 76 | 76 | 72 | |
| Mem0Executor=Gemini2026.05 | 77.67 | 82 | 82 | 80 | 80 | 74 | 68 | |
| Gemini-1.5-proModel Type=Proprietary MLLM, Enhancement=BrainMem2026.03 | 75.3 | 88 | 80 | 68 | 80 | 72 | 64 | |
| GPT-4oModel Type=Proprietary MLLM, Enhancement=BrainMem2026.03 | 75 | 82 | 82 | 64 | 78 | 72 | 72 | |
| Claude-3.5-SonnetModel Type=Proprietary MLLM, Enhancement=BrainMem2026.03 | 74.7 | 78 | 84 | 66 | 72 | 64 | 84 | |
| ELITEModel Category=Training-based MLLMs, Base Model=Qwen2.5-VL-72B-Ins2026.03 | 70.8 | 78 | 78 | 68 | 68 | 62 | — | |
| Claude-3.7-SonnetBase Model=Claude-3.7-Sonnet, Evaluation Protocol=Zero-Shot2026.04 | 67.7 | 68 | 70 | 68 | 68 | 62 | 70 | |
| Claude-3.7-SonnetModel Type=Proprietary MLLM, Enhancement=None2026.03 | 67.7 | 68 | 70 | 68 | 68 | 62 | 70 | |
| RoboAgentBase Model=Qwen2.5-VL-3B2026.04 | 67 | 72 | 64 | 78 | 48 | 60 | 80 | |
| Claude-3.5-SonnetModel Category=Proprietary MLLMs2026.03 | 66.4 | 72 | 76 | 60 | 66 | 58 | — | |
| ERA (3B)Model Category=Training-based MLLMs, Parameters=3B2026.03 | 65.2 | 72 | 72 | 62 | 54 | 66 | — | |
| GPT-4.1Type=Closed-Source General Model, Params=-2025.10 | 64.67 | 70 | 70 | 62 | 64 | 62 | 60 | |
| Claude-3.5-SonnetModel Type=Proprietary MLLM, Enhancement=None2026.03 | 64 | 72 | 76 | 60 | 66 | 58 | 52 | |
| Gemini-1.5-ProModel Category=Proprietary MLLMs2026.03 | 63.2 | 70 | 72 | 58 | 64 | 52 | — | |
| WAPBase Model=Qwen2.5-VL-7B2026.04 | 62.7 | 66 | 70 | 56 | 62 | 52 | 70 | |
| Gemini-1.5-ProBase Model=Gemini-1.5-Pro, Evaluation Protocol=Zero-Shot2026.04 | 62.3 | 70 | 72 | 58 | 64 | 52 | 58 | |
| Gemini-1.5-ProModel Type=Proprietary MLLM, Enhancement=None2026.03 | 62.3 | 70 | 72 | 58 | 64 | 52 | 58 | |
| No MemExecutor=Gemini2026.05 | 59 | 70 | 60 | 68 | 56 | 56 | 44 | |
| GPT-4oModel Category=Proprietary MLLMs2026.03 | 56.8 | 64 | 68 | 46 | 54 | 52 | — | |
| GPT-4oBase Model=GPT-4o, Evaluation Protocol=Zero-Shot2026.04 | 56.3 | 64 | 68 | 46 | 54 | 52 | 54 | |
| GPT-4oModel Type=Proprietary MLLM, Enhancement=None2026.03 | 56.3 | 64 | 68 | 46 | 54 | 52 | 54 | |
| RoboGPT-R1Type=Embodied Planning Model, Params=3B2025.10 | 55.33 | 62 | 64 | 50 | 56 | 50 | 50 | |
| RoboGPT-R1Base Model=Qwen2.5-VL-3B2026.04 | 55.3 | 62 | 64 | 50 | 56 | 50 | 50 | |
| VAGEN (3B)Model Category=Training-based MLLMs, Parameters=3B2026.03 | 52.8 | 70 | 70 | 58 | 38 | 28 | — | |
| Gemini-2.0-flashType=Closed-Source General Model, Params=-2025.10 | 52.3 | 62 | 54 | 46 | 48 | 46 | 58 | |
| GPT-4oType=Closed-Source General Model, Params=-2025.10 | 51.67 | 54 | 58 | 52 | 46 | 52 | 48 | |
| Gemini-2.0-flashModel Category=Proprietary MLLMs2026.03 | 51.2 | 62 | 54 | 46 | 48 | 46 | — | |
| RL4VLM (3B)Model Category=Training-based MLLMs, Parameters=3B2026.03 | 51.2 | 70 | 70 | 56 | 32 | 28 | — | |
| VlaserBase Model=InternVL3-8B2026.04 | 50 | — | — | — | — | — | — | |
| Qwen2.5-VL-72B-InsModel Category=Open-Source MLLMs2026.03 | 49.8 | 55 | 55 | 50 | 42 | 47 | — | |
| LangMemExecutor=GPT-52026.05 | 46.67 | 74 | 62 | 46 | 56 | 38 | 4 | |
| A-MemExecutor=GPT-52026.05 | 46.33 | 52 | 48 | 34 | 54 | 42 | 48 | |
| G-MemExecutor=GPT-52026.05 | 45.33 | 46 | 60 | 42 | 58 | 42 | 24 | |
| Qwen2.5-VL-72B-InsType=Open-Source General Model, Params=72B2025.10 | 43.67 | 62 | 48 | 40 | 36 | 44 | 32 | |
| Qwen-VL-MaxModel Category=Proprietary MLLMs2026.03 | 43.2 | 44 | 44 | 42 | 48 | 38 | — | |
| Mem0Executor=GPT-52026.05 | 43.15 | 48 | 60 | 47 | 58 | 36 | 10 | |
| Reinforced Reasoner (7B)Model Category=Training-based MLLMs, Parameters=7B2026.03 | 41.6 | 54 | 46 | 28 | 42 | 38 | — | |
| Qwen-VL-MaxBase Model=Qwen-VL-Max, Evaluation Protocol=Zero-Shot2026.04 | 41.3 | 44 | 44 | 42 | 48 | 38 | 32 | |
| Qwen-VL-MaxModel Type=Proprietary MLLM, Enhancement=None2026.03 | 41.3 | 44 | 44 | 42 | 48 | 38 | 32 | |
| Qwen-VL-MaxType=Closed-Source General Model, Params=-2025.10 | 41.3 | 44 | 44 | 42 | 48 | 38 | 32 | |
| MemCompilerExecutor=Qwen-3-VL-32B2026.05 | 40 | 52 | 48 | 38 | 54 | 38 | 10 | |
| Qwen2.5-VL-72BBase Model=Qwen2.5-VL-72B, Evaluation Protocol=Zero-Shot2026.04 | 39.7 | 50 | 42 | 36 | 42 | 34 | 34 | |
| Qwen2.5-VL-72BModel Type=Open-Source MLLM, Enhancement=None2026.03 | 39.7 | 50 | 42 | 36 | 42 | 34 | 34 | |
| InternVL3-78BModel Category=Open-Source MLLMs2026.03 | 39.6 | 38 | 46 | 42 | 34 | 38 | — | |
| Gemini-1.5-flashModel Type=Proprietary MLLM, Enhancement=None2026.03 | 39.3 | 44 | 56 | 42 | 40 | 26 | 28 | |
| gemma-3-27b-itModel Category=Open-Source MLLMs2026.03 | 39.2 | 42 | 48 | 30 | 40 | 36 | — | |
| InternVL3-78BModel Type=Open-Source MLLM, Enhancement=None2026.03 | 39 | 38 | 46 | 42 | 34 | 38 | 36 | |
| LoHo-Manip-4BParameters=4B2026.04 | 38 | 48 | 54 | 28 | 36 | 30 | 31 | |
| InternVL2_5-78BModel Type=Open-Source MLLM, Enhancement=None2026.03 | 37.7 | 38 | 42 | 34 | 34 | 36 | 42 | |
| InternVL2 5-78BModel Category=Open-Source MLLMs2026.03 | 36.8 | 38 | 42 | 34 | 34 | 36 | — | |
| REBPBase Model=Qwen2.5-VL-7B2026.04 | 35.6 | 54 | 46 | 28 | 42 | 38 | 6 | |
| MemCompilerExecutor=Qwen-2.5-VL-32B2026.05 | 35.33 | 42 | 44 | 38 | 50 | 34 | 4 | |
| Llama-3.2-90B-VisionModel Category=Open-Source MLLMs2026.03 | 35.2 | 38 | 44 | 28 | 34 | 32 | — | |
| REBPType=Embodied Planning Model, Params=7B2025.10 | 35 | 52 | 46 | 28 | 46 | 32 | 6 | |
| Qwen2-VL-72B-InsModel Category=Open-Source MLLMs2026.03 | 34.4 | 40 | 40 | 30 | 30 | 32 | — | |
| GPT-4o-miniType=Closed-Source General Model, Params=-2025.10 | 34 | 66 | 68 | 14 | 56 | 0 | 0 | |
| A-MemExecutor=Qwen-3-VL-32B2026.05 | 32.33 | 46 | 34 | 30 | 50 | 30 | 4 | |
| Llama-3.2-90B-VisionModel Type=Open-Source MLLM, Enhancement=None2026.03 | 32 | 38 | 44 | 28 | 34 | 32 | 16 | |
| No MemExecutor=GPT-52026.05 | 32 | 60 | 28 | 36 | 40 | 28 | 0 | |
| Llama-3.2-90B-Vision-InsType=Open-Source General Model, Params=90B2025.10 | 32 | 38 | 44 | 28 | 34 | 32 | 16 | |
| LangmemExecutor=Qwen-3-VL-32B2026.05 | 31.33 | 38 | 44 | 28 | 44 | 26 | 8 | |
| Mem0Executor=Qwen-3-VL-32B2026.05 | 30.33 | 40 | 40 | 26 | 42 | 26 | 8 | |
| Ovis2-34BModel Category=Open-Source MLLMs2026.03 | 29.6 | 34 | 38 | 28 | 30 | 18 | — | |
| Ovis2-34BModel Type=Open-Source MLLM, Enhancement=None2026.03 | 28.7 | 34 | 38 | 28 | 30 | 18 | 24 | |
| Gemma-3-12b-itsType=Open-Source General Model, Params=12B2025.10 | 25.7 | 32 | 38 | 26 | 26 | 20 | 12 | |
| G-MemExecutor=Qwen-3-VL-32B2026.05 | 25.33 | 34 | 38 | 24 | 36 | 14 | 6 | |
| GPT-4o-miniModel Type=Proprietary MLLM, Enhancement=None2026.03 | 24 | 34 | 36 | 24 | 28 | 22 | 0 | |
| GPT-4o miniSize=mini2026.04 | 24 | 34 | 36 | 24 | 28 | 22 | 0 | |
| Qwen3-VL-4BParameters=4B2026.04 | 19 | 18 | 26 | 14 | 14 | 14 | 26 | |
| No MemExecutor=Qwen-3-VL-32B2026.05 | 19 | 24 | 28 | 24 | 24 | 14 | 0 | |
| No MemExecutor=Qwen-2.5-VL-32B2026.05 | 18 | 22 | 24 | 12 | 26 | 22 | 2 | |
| Mem0Executor=Qwen-2.5-VL-32B2026.05 | 14.66 | 20 | 26 | 12 | 18 | 10 | 2 | |
| LangmemExecutor=Qwen-2.5-VL-32B2026.05 | 14.33 | 22 | 26 | 6 | 20 | 12 | 0 | |
| MemGenExecutor=Qwen-2.5-VL-32B2026.05 | 14.33 | 14 | 16 | 6 | 10 | 8 | 0 | |
| MemGenExecutor=Qwen-3-VL-32B2026.05 | 14.22 | 22 | 16 | 6 | 18 | 20 | 4 | |
| RoboBrain2.0Base Model=Qwen2.5-VL-7B2026.04 | 14 | — | — | — | — | — | — | |
| Llama3.2-11BParameters=11B2026.04 | 14 | 24 | 16 | 22 | 8 | 6 | 6 | |
| InternVL3-8BParameters=8B2026.04 | 10 | 20 | 14 | 12 | 14 | 0 | 2 | |
| G-MemExecutor=Qwen-2.5-VL-32B2026.05 | 7.67 | 4 | 6 | 16 | 14 | 6 | 0 | |
| A-MemExecutor=Qwen-2.5-VL-32B2026.05 | 3 | 2 | 4 | 6 | 4 | 2 | 0 | |
| Qwen2.5-VL-7B-InsType=Open-Source General Model, Params=7B2025.10 | 2.67 | 6 | 6 | 0 | 2 | 2 | 0 | |
| InternVL2.5-8BType=Open-Source General Model, Params=8B2025.10 | 2 | 4 | 2 | 0 | 6 | 0 | 0 | |
| Qwen2.5-VL-3B-InsType=Open-Source General Model, Params=3B2025.10 | 1.33 | 2 | 0 | 0 | 2 | 4 | 0 | |
| RoboBrainType=Embodied Planning Model, Params=7B2025.10 | 0.33 | 2 | 0 | 0 | 0 | 0 | 0 | |
| TapaType=Embodied Planning Model, Params=7B2025.10 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |