Dialogue Memory Accuracy on LongMemEval-S (N=500)
91Temporal AccuracyHindsight
Evaluation Results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| HindsightGenerator Backbone=Gemini-3 Pro, Memory Architecture Backbone=GPT-OSS-120b2025.12 | 91 | 87.2 | 94.9 | 97.1 | 96.4 | 80 | 91.4 | — | |
| HindsightBackbone=OSS-120B2025.12 | 85.7 | 81.2 | 92.3 | 100 | 98.2 | 86.7 | 89 | — | |
| SupermemoryBackbone=Gemini-3 Pro2025.12 | 82 | 76.7 | 89.7 | 98.6 | 98.2 | 70 | 85.2 | — | |
| SupermemoryBackbone=GPT-52025.12 | 81.2 | 75.2 | 87.2 | 97.1 | 100 | 76.7 | 84.6 | — | |
| HindsightBackbone=OSS-20B2025.12 | 79.7 | 79.7 | 84.6 | 95.7 | 94.6 | 66.7 | 83.6 | — | |
| SupermemoryBackbone=GPT-4o2025.12 | 76.7 | 71.4 | 88.5 | 97.1 | 96.4 | 70 | 81.6 | — | |
| LightMemLLM Judge=GPT-4o-mini2026.01 | 67.18 | 71.74 | 83.12 | 87.14 | 32.14 | 68.18 | 69.81 | — | |
| ES-MemLLM Judge=GPT-4o-mini2026.01 | 64.66 | 66.17 | 78.21 | 84.29 | 82.14 | 73.33 | 72.4 | — | |
| ZepBackbone=GPT-4o2025.12 | 62.4 | 57.9 | 83.3 | 92.9 | 80.4 | 56.7 | 71.2 | — | |
| HeLa-MemBackbone=GPT-4o-mini, LLM judge=GPT-4o-mini, Total retrieval budget=202026.04 | 50.38 | 57.14 | 78.21 | — | — | — | — | 78.85 | |
| A-MemLLM Judge=GPT-4o-mini2026.01 | 47.36 | 48.87 | 64.11 | 92.86 | 96.43 | 46.67 | 62.2 | — | |
| A-MEMBackbone=GPT-4o-mini, LLM judge=GPT-4o-mini, Total retrieval budget=202026.04 | 47.36 | 48.87 | 64.11 | — | — | — | — | 84.62 | |
| Full-contextBackbone=GPT-4o2025.12 | 45.1 | 44.3 | 78.2 | 81.4 | 94.6 | 20 | 60.2 | — | |
| Mem0LLM Judge=GPT-4o-mini2026.01 | 40.15 | 46.21 | 70.12 | 81.43 | 41.07 | 60 | 53.51 | — | |
| Mem0Backbone=GPT-4o-mini, LLM judge=GPT-4o-mini, Total retrieval budget=202026.04 | 40.15 | 46.21 | 70.12 | — | — | — | — | 62.82 | |
| Naive RAGLLM Judge=GPT-4o-mini2026.01 | 39.85 | 48.48 | 67.95 | 90 | 98.21 | 53.33 | 60.9 | — | |
| NaiveRAGBackbone=GPT-4o-mini, LLM judge=GPT-4o-mini, Total retrieval budget=202026.04 | 39.85 | 48.48 | 67.95 | — | — | — | — | 85.9 | |
| MemoryOSLLM Judge=GPT-4o-mini2026.01 | 32.33 | 31.06 | 48.72 | 80 | 64.29 | 30 | 44.66 | — | |
| MemoryOSBackbone=GPT-4o-mini, LLM judge=GPT-4o-mini, Total retrieval budget=202026.04 | 32.33 | 31.06 | 48.72 | — | — | — | — | 64.74 | |
| Full-contextBackbone=OSS-20B2025.12 | 31.6 | 21.1 | 60.3 | 38.6 | 80.4 | 20 | 39 | — | |
| Full TextLLM Judge=GPT-4o-mini2026.01 | 31.58 | 45.45 | 76.92 | 87.14 | 89.29 | 36.67 | 56.89 | — | |
| FullTextBackbone=GPT-4o-mini, LLM judge=GPT-4o-mini, Total retrieval budget=202026.04 | 31.58 | 45.45 | 76.92 | — | — | — | — | 78.21 | |
| LangMemLLM Judge=GPT-4o-mini2026.01 | 15.79 | 20.3 | 66.67 | 60 | 46.43 | 60 | 37.2 | — | |
| LangMemBackbone=GPT-4o-mini, LLM judge=GPT-4o-mini, Total retrieval budget=202026.04 | 15.79 | 20.3 | 66.67 | — | — | — | — | 55.13 |