Question Answering on LoCoMo-10
81.91Multi-hop Accuracy (Cat1)DimMem
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| DimMemQA Model=GPT-4.1-mini2026.05 | 81.91 | 81.62 | 55.21 | 82.5 | 80.51 | 3,859 | |
| DA-UpdateQA Model=GPT-4.1-mini2026.05 | 80.85 | 81 | 55.21 | 84.78 | 81.43 | — | |
| DimMem-4BQA Model=GPT-4.1-mini, Memory Extractor=fine-tuned Qwen3-4B2026.05 | 76.95 | 81.31 | 57.29 | 82.88 | 79.87 | — | |
| DimMemQA Model=Qwen3-30B-A3B-Instruct-25072026.05 | 75.53 | 69.16 | 58.33 | 81.09 | 76.17 | 5,846 | |
| DimMem-4B†QA Model=Qwen3-4B, Memory Extractor=fine-tuned Qwen3-4B2026.05 | 72.7 | 75.7 | 46.88 | 76.1 | 73.57 | — | |
| GAAMALLM=GPT-4o-mini, Embeddings=text-embedding-3-small, Context budget=1,000-word2026.03 | 72.2 | 71.9 | 49.3 | 87.2 | 78.9 | — | |
| MemOSQA Model=GPT-4.1-mini2026.05 | 70.57 | 40.19 | 48.96 | 82.28 | 69.29 | 7,457 | |
| SimpleMemQA Model=GPT-4.1-mini2026.05 | 68.44 | 63.86 | 59.38 | 86.44 | 76.75 | 6,527 | |
| RAG BaselineLLM=GPT-4o-mini, Embeddings=text-embedding-3-small, Context budget=1,000-word2026.03 | 67.5 | 59 | 44.6 | 87.1 | 75 | — | |
| SimpleMemQA Model=Qwen3-30B-A3B-Instruct-25072026.05 | 64.54 | 62.31 | 56.25 | 80.26 | 72.14 | 6,993 | |
| A-MemQA Model=GPT-4.1-mini2026.05 | 63.12 | 40.81 | 50 | 78 | 65.78 | 9,743 | |
| MemoryOSQA Model=GPT-4.1-mini2026.05 | 62.77 | 39.25 | 43.75 | 76.69 | 64.28 | 8,798 | |
| LightMemQA Model=GPT-4.1-mini2026.05 | 62.77 | 64.17 | 54.17 | 81.57 | 72.79 | 5,063 | |
| HippoRAGLLM=GPT-4o-mini2026.03 | 61.7 | 67 | 67.7 | 74.1 | 69.9 | — | |
| LightMemQA Model=Qwen3-30B-A3B-Instruct-25072026.05 | 61.7 | 60.75 | 51.04 | 79.31 | 70.45 | 6,721 | |
| NaiveRAGQA Model=GPT-4.1-mini2026.05 | 61.35 | 37.07 | 41.67 | 74.67 | 62.34 | 10,234 | |
| MemOSQA Model=Qwen3-30B-A3B-Instruct-25072026.05 | 59.93 | 39.56 | 53.13 | 77.65 | 64.94 | 7,537 | |
| FullTextQA Model=GPT-4.1-mini2026.05 | 59.22 | 68.22 | 50 | 72.41 | 67.72 | — | |
| NaiveRAGQA Model=Qwen3-30B-A3B-Instruct-25072026.05 | 57.45 | 40.19 | 55.21 | 77.29 | 64.55 | 11,243 | |
| FullTextQA Model=Qwen3-30B-A3B-Instruct-25072026.05 | 56.74 | 46.73 | 57.29 | 87.75 | 71.62 | — | |
| MemoryOSQA Model=Qwen3-30B-A3B-Instruct-25072026.05 | 56.03 | 42.99 | 47.92 | 69.32 | 60.06 | 8,537 | |
| A-MemQA Model=Qwen3-30B-A3B-Instruct-25072026.05 | 53.19 | 41.12 | 55.21 | 66.11 | 57.86 | 10,347 | |
| NemoriLLM=GPT-4o-mini2026.03 | 49.4 | 45 | 36.8 | 57.4 | 52.1 | — | |
| A-MemLLM=GPT-4o-mini2026.03 | 44.7 | 37.4 | 50 | 51.5 | 47.2 | — |