Question Answering on LoCoMo (Detailed Hop Metrics)
67.13Single Hop F1Mem0
Evaluation results
| Method | Links | ||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Mem0Base Model=gpt-4.1-nano, Reasoning Level=medium2026.02 | 67.13 | — | 51.15 | — | 55.51 | — | 72.93 | — | — | — | — | — | — | — | — | — | |
| MemCoTModel=GPT-4.1-mini, Token=2,7902026.04 | 66.42 | — | 43.89 | — | 60.88 | — | 40.88 | — | 59.54 | — | — | — | — | — | — | — | |
| Mem0gBase Model=gpt-4.1-nano, Reasoning Level=medium, Graph memory enhancement=true2026.02 | 65.71 | — | 47.19 | — | 58.13 | — | 75.71 | — | — | — | — | — | — | — | — | — | |
| MemCoT (w/o ψ_z-o)Model=GPT-4.1-mini, Token=1,7072026.04 | 64.58 | — | 38.51 | — | 38.94 | — | 52.81 | — | 53.73 | — | — | — | — | — | — | — | |
| MemCoTModel=Qwen2.5-7B2026.04 | 63.99 | — | 36.34 | — | 40.06 | — | 33.08 | — | 52.01 | — | — | — | — | — | — | — | |
| OpenAIBase Model=gpt-4.1-nano, Reasoning Level=medium2026.02 | 63.79 | — | 42.92 | — | 21.71 | — | 62.29 | — | — | — | — | — | — | — | — | — | |
| OpenSageBase Model=gpt-4.1-nano, Reasoning Level=medium2026.02 | 63.21 | — | 45.89 | — | 57.84 | — | 76.38 | — | — | — | — | — | — | — | — | — | |
| A-Mem*Base Model=gpt-4.1-nano, Reasoning Level=medium2026.02 | 62.23 | — | 47.92 | — | 23.43 | — | 71.12 | — | — | — | — | — | — | — | — | — | |
| ZepBase Model=gpt-4.1-nano, Reasoning Level=medium2026.02 | 61.7 | — | 41.35 | — | 49.31 | — | 76.6 | — | — | — | — | — | — | — | — | — | |
| Bi-MemBackbone=GPT-4o-mini2026.01 | 53.68 | 47.99 | 39.17 | 31.81 | 54.44 | 41.56 | 30.99 | 26.9 | 49.74 | 42.33 | — | — | — | — | — | — | |
| RAGBackbone=GPT-4o-mini2026.01 | 52.45 | 47.94 | 27.5 | 20.13 | 46.07 | 40.35 | 23.23 | 17.94 | 44.67 | 39.33 | — | — | — | — | — | — | |
| CAMBackbone=GPT-4o-mini2026.01 | 49.94 | 46.68 | 36.63 | 25.37 | 18.85 | 12.9 | 22.71 | 18.93 | 39.25 | 33.92 | — | — | — | — | — | — | |
| Bi-MemBackbone=Qwen2.5-14B-Instruct2026.01 | 48.07 | 42.95 | 32.42 | 25.53 | 44.21 | 29.29 | 18.38 | 16.11 | 42.51 | 35.19 | — | — | — | — | — | — | |
| RAGBackbone=Qwen2.5-14B-Instruct2026.01 | 47.87 | 42.79 | 26.38 | 19.54 | 30.78 | 25.97 | 14.16 | 10.52 | 38.2 | 32.95 | — | — | — | — | — | — | |
| MEM0Backbone=GPT-4o-mini2026.01 | 47.65 | 38.72 | 38.72 | 27.13 | 48.93 | 40.51 | 28.64 | 21.58 | 45.08 | 35.88 | — | — | — | — | — | — | |
| A-MEMBackbone=GPT-4o-mini2026.01 | 44.65 | 37.06 | 27.02 | 20.09 | 45.85 | 36.67 | 12.14 | 12 | 39.61 | 32.27 | — | — | — | — | — | — | |
| HeLa-MemBackbone=GPT-4o-mini, GPT-4o, Qwen2.5-3b (Averaged)2026.04 | 43.7 | 38.75 | 33.13 | 25.22 | 40.96 | 35.72 | 22.11 | 17.64 | — | — | — | — | — | — | — | 1.25 | |
| MEM0Backbone=Qwen2.5-14B-Instruct2026.01 | 42.58 | 35.15 | 31.73 | 24.82 | 28.96 | 26.24 | 15.03 | 11.28 | 35.99 | 29.88 | — | — | — | — | — | — | |
| LIGHTMEMBackbone=GPT-4o-mini2026.01 | 41.79 | 37.83 | 29.78 | 24.8 | 43.71 | 39.72 | 16.89 | 13.92 | 38.41 | 34.32 | — | — | — | — | — | — | |
| Mem0Model=GPT-4.1-mini, Token=1,7642026.04 | 41.3 | — | 30.14 | — | 48.91 | — | 16.43 | — | 34.2 | — | — | — | — | — | — | — | |
| A-MemModel=GPT-4.1-mini, Token=2,5202026.04 | 41.02 | — | 25.06 | — | 51.01 | — | 13.22 | — | 32.58 | — | — | — | — | — | — | — | |
| ZepModel=GPT-4.1-mini, Token=1,6022026.04 | 39.96 | — | 26.73 | — | 20.97 | — | 21.24 | — | 32.4 | — | — | — | — | — | — | — | |
| A-MemBackbone=GPT-4o-mini, GPT-4o, Qwen2.5-3b (Averaged)2026.04 | 38.9 | 33.09 | 26.04 | 18.6 | 36.53 | 29.21 | 15.24 | 14.05 | — | — | — | — | — | — | — | 3 | |
| MemoryOSBackbone=GPT-4o-mini, GPT-4o, Qwen2.5-3b (Averaged), reproduced=true2026.04 | 38.45 | 32.84 | 32.61 | 25.42 | 35.33 | 28.53 | 19.3 | 14.63 | — | — | — | — | — | — | — | 2.25 | |
| LoCoMoBackbone=GPT-4o-mini, GPT-4o, Qwen2.5-3b (Averaged)2026.04 | 36.32 | 29.64 | 19.21 | 14.17 | 10.2 | 7.75 | 11.02 | 10.64 | — | — | — | — | — | — | — | 5 | |
| MemGPTBackbone=GPT-4o-mini, GPT-4o, Qwen2.5-3b (Averaged)2026.04 | 36.15 | 31.07 | 20.69 | 14.95 | 15.25 | 11.86 | 9.48 | 8.8 | — | — | — | — | — | — | — | 4.5 | |
| LIGHTMEMBackbone=Qwen2.5-14B-Instruct2026.01 | 34.92 | 31.22 | 25.45 | 19.61 | 32.03 | 27.7 | 15.81 | 11.81 | 31.37 | 27.12 | — | — | — | — | — | — | |
| A-MEMBackbone=Qwen2.5-14B-Instruct2026.01 | 33.75 | 30.04 | 22.09 | 15.28 | 27.19 | 22.05 | 13.49 | 10.74 | 28.95 | 24.43 | — | — | — | — | — | — | |
| LONGCONTEXTBackbone=GPT-4o-mini2026.01 | 32.69 | 20.85 | 22.87 | 16.84 | 12.47 | 8.86 | 7.87 | 5.89 | 25.08 | 16.65 | — | — | — | — | — | — | |
| CAMBackbone=Qwen2.5-14B-Instruct2026.01 | 32.69 | 23.15 | 24.78 | 18.98 | 5.21 | 2.71 | 15.25 | 12.45 | 24.37 | 17.42 | — | — | — | — | — | — | |
| MemoryOSModel=Qwen2.5-7B2026.04 | 29.55 | — | 21.03 | — | 26.26 | — | 40.85 | — | 33.74 | — | — | — | — | — | — | — | |
| MemVerseModel=Qwen2.5-7B2026.04 | 28.12 | — | 24.23 | — | 25.05 | — | 40.33 | — | 33.79 | — | — | — | — | — | — | — | |
| MemGPTModel=GPT-4.1-mini, Token=16,9772026.04 | 25.59 | — | 17.72 | — | 19.44 | — | 11.29 | — | 18.51 | — | — | — | — | — | — | — | |
| Mem0Model=Qwen2.5-7B2026.04 | 24.96 | — | 20.31 | — | 33.16 | — | 32.74 | — | 28.75 | — | — | — | — | — | — | — | |
| SECOMBackbone=Qwen2.5-14B-Instruct2026.01 | 22.26 | 18.71 | 19.15 | 14.26 | 19.58 | 16.3 | 15.75 | 13.01 | 20.71 | 17.02 | — | — | — | — | — | — | |
| RAGModel=Qwen2.5-7B2026.04 | 21.72 | — | 8.05 | — | 4.89 | — | 14.51 | — | 13.88 | — | — | — | — | — | — | — | |
| LONGCONTEXTBackbone=Qwen2.5-14B-Instruct2026.01 | 21 | 13.17 | 18.09 | 13.21 | 10.54 | 5.82 | 7.31 | 4.71 | 17.41 | 11.1 | — | — | — | — | — | — | |
| Full ContextModel=GPT-4.1-mini, Token=16,9102026.04 | 18.68 | — | 25.02 | — | 12.04 | — | 19.05 | — | 18.7 | — | — | — | — | — | — | — | |
| EpMemModel=Qwen2.5-7B2026.04 | 17.64 | — | 6.35 | — | 9.96 | — | 2.66 | — | 6.63 | — | — | — | — | — | — | — | |
| SECOMBackbone=GPT-4o-mini2026.01 | 16.67 | 13.96 | 15.06 | 11.93 | 14.58 | 12.2 | 17.29 | 14.76 | 15.97 | 13.25 | — | — | — | — | — | — | |
| A-MemModel=Qwen2.5-7B2026.04 | 12.73 | — | 8.67 | — | 6.28 | — | 13.86 | — | 12.1 | — | — | — | — | — | — | — | |
| Full ContextModel=Qwen2.5-7B2026.04 | 10.61 | — | 3.39 | — | 3.72 | — | 8.24 | — | 7.38 | — | — | — | — | — | — | — | |
| ReadAgentBackbone=GPT-4o-mini, GPT-4o, Qwen2.5-3b (Averaged)2026.04 | 8.46 | 6.82 | 8.74 | 6.07 | 6.59 | 5.02 | 6.57 | 6.24 | — | — | — | — | — | — | — | 6.38 | |
| MemoryBankBackbone=GPT-4o-mini, GPT-4o, Qwen2.5-3b (Averaged)2026.04 | 6.33 | 5.19 | 5.03 | 4.28 | 4.62 | 3.8 | 6.21 | 5.94 | — | — | — | — | — | — | — | 6.88 | |
| MemoryBankModel=GPT-4.1-mini, Token=4322026.04 | 5.72 | — | 5 | — | 5.94 | — | 5.16 | — | 5.46 | — | — | — | — | — | — | — | |
| ENGRAMSource=Patel and Patel [2025]2026.03 | — | — | — | — | — | — | — | — | — | — | 23.1 | 18.3 | 21.9 | 8.6 | 21.1 | — | |
| KumihoSource=This work2026.03 | — | — | — | — | — | — | — | — | — | — | 46.2 | 35.5 | 53.3 | 29 | 44.7 | — | |
| Mem0Source=Zep [2025]2026.03 | — | — | — | — | — | — | — | — | — | — | 38.7 | 28.6 | 48.9 | 47.7 | 40 | — | |
| Mem0-GraphSource=Zep [2025]2026.03 | — | — | — | — | — | — | — | — | — | — | 38.1 | 24.3 | 51.6 | 49.3 | 40 | — | |
| MemobaseSource=Memobase [2025]2026.03 | — | — | — | — | — | — | — | — | — | — | 46.3 | 22.9 | 64.2 | 51.6 | — | — | |
| OpenAI MemSource=Zep [2025]2026.03 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 34 | — | |
| ZepSource=Zep [2025]2026.03 | — | — | — | — | — | — | — | — | — | — | 35.7 | 19.4 | 42 | 49.6 | — | — |