Question Answering on LongMemEval S (test)
84.21QA Score (TR Context)LightMem
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| LightMemBackbone=GPT-4.1-mini2026.05 | 84.21 | 51.13 | 88.46 | 90 | 23.21 | 76.67 | 69.6 | |
| SimpleMemBackbone=GPT-4.1-mini2026.05 | 81.95 | 57.89 | 74.36 | 82.86 | 71.43 | 76.67 | 73 | |
| DA-UpdateBackbone=GPT-4.1-mini2026.05 | 77.44 | 69.17 | 88.46 | 98.57 | 73.21 | 56.67 | 78.2 | |
| DimMemBackbone=Qwen3-30B-A3B-Instruct-25072026.05 | 76.69 | 54.89 | 79.49 | 94.29 | 73.21 | 56.67 | 72.2 | |
| DimMem-4B†Extractor=Qwen3-4B, QA Model=Qwen3-4B2026.05 | 76.69 | 55.64 | 83.33 | 84.29 | 42.86 | 56.67 | 68.2 | |
| LightMemBackbone=Qwen3-30B-A3B-Instruct-25072026.05 | 75.94 | 57.89 | 89.74 | 85.71 | 21.43 | 73.33 | 68.4 | |
| DimMemBackbone=GPT-4.1-mini2026.05 | 74.44 | 64.66 | 84.62 | 98.57 | 76.79 | 63.33 | 76.4 | |
| SimpleMemBackbone=Qwen3-30B-A3B-Instruct-25072026.05 | 74.44 | 52.63 | 75.64 | 87.14 | 64.29 | 66.67 | 69 | |
| DimMem-4BExtractor=Qwen3-4B, QA Model=GPT-4.1-mini2026.05 | 73.68 | 67.42 | 83.33 | 90 | 46.43 | 60 | 71.8 | |
| MemOSBackbone=Qwen3-30B-A3B-Instruct-25072026.05 | 72.93 | 51.13 | 73.08 | 80 | 26.79 | 53.33 | 61.8 | |
| A-MemBackbone=Qwen3-30B-A3B-Instruct-25072026.05 | 66.92 | 44.36 | 71.79 | 74.29 | 35.71 | 63.33 | 59 | |
| NaiveRAGBackbone=Qwen3-30B-A3B-Instruct-25072026.05 | 65.41 | 48.87 | 71.79 | 81.43 | 30.36 | 53.33 | 59.6 | |
| MemOSBackbone=GPT-4.1-mini2026.05 | 63.91 | 54.14 | 70.51 | 90 | 64.29 | 50 | 65.2 | |
| FullTextBackbone=Qwen3-30B-A3B-Instruct-25072026.05 | 63.91 | 42.11 | 65.38 | 71.43 | 32.14 | 46.67 | 54.8 | |
| A-MemBackbone=GPT-4.1-mini2026.05 | 58.65 | 52.63 | 69.23 | 88.57 | 57.14 | 46.67 | 62 | |
| NaiveRAGBackbone=GPT-4.1-mini2026.05 | 57.14 | 54.14 | 74.36 | 77.14 | 60.71 | 53.33 | 62 | |
| MemoryOSBackbone=Qwen3-30B-A3B-Instruct-25072026.05 | 57.14 | 40.6 | 64.1 | 61.43 | 28.57 | 50 | 50.8 | |
| MemoryOSBackbone=GPT-4.1-mini2026.05 | 50.38 | 48.12 | 71.79 | 85.71 | 51.79 | 66.67 | 59.2 | |
| FullTextBackbone=GPT-4.1-mini2026.05 | 31.58 | 46.62 | 43.59 | 80 | 41.07 | 56.67 | 46.8 |