Long-term Dialogue on LoCoMo (test)
70.57Multi-Hop AccuracyLightMem
Evaluation Results
| Method | Links | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| LightMemBase Model=Qwen3-VL-Instruct (MLLM), Model Scale=32B, Latency (s)=700.502026.05 | 70.57 | 38.68 | 59.19 | 49.79 | 50 | 20.52 | 84.19 | 57.2 | 74.35 | 49.98 | |
| MemOSBase Model=Qwen3 (LLM), Model Scale=32B, Latency (s)=628.762026.05 | 69.86 | 41.31 | 65.11 | 32.99 | 54.17 | 29.94 | 85.26 | 57.66 | 76.3 | 47.79 | |
| MemOSBase Model=Qwen3-VL-Instruct (MLLM), Model Scale=32B, Latency (s)=669.452026.05 | 69.86 | 40.26 | 72.59 | 42.87 | 58.33 | 28.09 | 86.68 | 59.53 | 78.9 | 50.57 | |
| EverMemOSBase Model=Qwen3-VL-Instruct (MLLM), Model Scale=32B, Latency (s)=1957.942026.05 | 68.79 | 12.16 | 73.52 | 21.48 | 44.79 | 7.42 | 85.14 | 16 | 77.21 | 15.91 | |
| Text RAGBase Model=Qwen3-VL-Instruct (MLLM), Model Scale=32B, Latency (s)=2.102026.05 | 67.73 | 42.26 | 73.21 | 60.82 | 56.25 | 31.13 | 84.54 | 61.04 | 77.34 | 55.69 | |
| MemOSBase Model=Qwen3-VL-Instruct (MLLM), Model Scale=8B, Latency (s)=673.582026.05 | 67.38 | 44.68 | 45.17 | 39.2 | 50 | 25.74 | 83.71 | 61.33 | 70.58 | 51.45 | |
| EverMemOSBase Model=Qwen3-VL-Instruct (MLLM), Model Scale=8B, Latency (s)=929.832026.05 | 66.31 | 15.15 | 68.54 | 22.29 | 50 | 8.14 | 85.37 | 18.98 | 76.17 | 18.29 | |
| MemShotBase Model=Qwen3-VL-Instruct (MLLM), Model Scale=32B, Latency (s)=9.562026.05 | 65.96 | 40.62 | 78.19 | 63.14 | 53.12 | 30.85 | 87.75 | 65.75 | 79.61 | 58.43 | |
| MemOCRBase Model=Qwen3-VL-Instruct (MLLM), Model Scale=32B, Latency (s)=12.032026.05 | 65.6 | 29.58 | 67.91 | 57.85 | 62.5 | 30.05 | 78.36 | 40.03 | 72.86 | 41.21 | |
| MemoryOSBase Model=Qwen3-VL-Instruct (MLLM), Model Scale=32B, Latency (s)=3471.942026.05 | 64.54 | 41.43 | 37.69 | 49.76 | 48.36 | 26.5 | 76.22 | 56.52 | 64.35 | 50.48 | |
| Text RAGBase Model=Qwen3-VL-Instruct (MLLM), Model Scale=8B, Latency (s)=2.102026.05 | 61.7 | 40.24 | 58.57 | 55.68 | 41.67 | 23.87 | 81.09 | 60.09 | 70.39 | 53.28 | |
| MemShotBase Model=Qwen3-VL-Instruct (MLLM), Model Scale=8B, Latency (s)=9.562026.05 | 60.99 | 39.99 | 70.72 | 60.07 | 41.67 | 22.84 | 85.37 | 64.44 | 75.13 | 56.46 | |
| LightMemBase Model=Qwen3 (LLM), Model Scale=32B, Latency (s)=814.342026.05 | 60.28 | 37.75 | 55.14 | 31.77 | 51.04 | 21.76 | 80.02 | 49.64 | 69.42 | 42 | |
| Text RAGBase Model=Qwen3 (LLM), Model Scale=32B, Latency (s)=2.102026.05 | 59.22 | 27.41 | 67.91 | 35.81 | 37.5 | 16.8 | 78 | 36.81 | 69.94 | 33.63 | |
| EverMemOSBase Model=Qwen3 (LLM), Model Scale=32B, Latency (s)=1535.982026.05 | 58.87 | 12.62 | 62.31 | 20.2 | 51.04 | 9.3 | 73.01 | 16.94 | 66.82 | 16.35 | |
| LightMemBase Model=Qwen3-VL-Instruct (MLLM), Model Scale=8B, Latency (s)=439.062026.05 | 58.16 | 37.85 | 47.98 | 45.84 | 46.88 | 22.08 | 81.33 | 54.32 | 67.99 | 47.53 | |
| MemOSBase Model=Qwen3-VL-Instruct (MLLM), Model Scale=2B, Latency (s)=670.602026.05 | 57.8 | 37.28 | 28.97 | 33.8 | 38.54 | 17.94 | 76.22 | 54.34 | 60.65 | 44.67 | |
| LightMemBase Model=Qwen3-VL-Instruct (MLLM), Model Scale=2B, Latency (s)=676.032026.05 | 54.96 | 35.15 | 30.53 | 40.48 | 39.58 | 18.23 | 74.91 | 48.96 | 59.81 | 42.75 | |
| MemOCRBase Model=Qwen3-VL-Instruct (MLLM), Model Scale=8B, Latency (s)=5.312026.05 | 54.96 | 31.41 | 39.56 | 47.44 | 57.29 | 29.85 | 73.84 | 43.35 | 62.21 | 41.17 | |
| MemoryOSBase Model=Qwen3-VL-Instruct (MLLM), Model Scale=8B, Latency (s)=1203.242026.05 | 53.9 | 36.17 | 26.48 | 39.66 | 42.71 | 20.89 | 72.77 | 53.41 | 57.79 | 45.36 | |
| MemoryOSBase Model=Qwen3 (LLM), Model Scale=32B, Latency (s)=2594.092026.05 | 53.55 | 35.65 | 29.91 | 39.53 | 51.04 | 25.72 | 74.44 | 54.18 | 59.87 | 45.96 | |
| MemShotBase Model=Qwen3-VL-Instruct (MLLM), Model Scale=2B, Latency (s)=9.562026.05 | 52.13 | 34.03 | 37.07 | 47.21 | 40.62 | 18.06 | 81.81 | 57.57 | 64.48 | 48.64 | |
| Text RAGBase Model=Qwen3-VL-Instruct (MLLM), Model Scale=2B, Latency (s)=2.102026.05 | 47.16 | 31.31 | 28.66 | 39.18 | 36.46 | 14.56 | 69.44 | 51.08 | 54.81 | 42.7 | |
| MemoryOSBase Model=Qwen3-VL-Instruct (MLLM), Model Scale=2B, Latency (s)=571.852026.05 | 39.01 | 15.69 | 12.77 | 34.51 | 32.29 | 7.8 | 58.5 | 23.92 | 43.77 | 23.62 | |
| EverMemOSBase Model=Qwen3-VL-Instruct (MLLM), Model Scale=2B, Latency (s)=1670.102026.05 | 37.23 | 8.68 | 25.23 | 16.69 | 29.17 | 4.47 | 60.05 | 11.5 | 46.69 | 11.63 | |
| MemOCRBase Model=Qwen3-VL-Instruct (MLLM), Model Scale=2B, Latency (s)=3.432026.05 | 28.72 | 26.31 | 15.26 | 25.05 | 37.5 | 21.4 | 40.43 | 37.54 | 32.86 | 31.87 |