Long-term Dialogue Understanding on LongMemEval (test)
91.43SS-User AccuracyMemShot
Evaluation results
| Method | Links | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| MemShotBackbone Model=Qwen3-VL-2B-Instruct2026.05 | 91.43 | 75.38 | 91.07 | 76.03 | 30 | 6.5 | 23.31 | 21.07 | 48.72 | 37.34 | 26.32 | 27.65 | 45.6 | 38.24 | |
| MemOSBackbone Model=Qwen3-VL-2B-Instruct2026.05 | 80 | 25.17 | 92.86 | 27.87 | 53.33 | 14.19 | 29.32 | 6.22 | 56.41 | 12.72 | 33.83 | 8.61 | 50.4 | 13.55 | |
| Text RAGBackbone Model=Qwen3-VL-2B-Instruct2026.05 | 77.14 | 37.68 | 80.36 | 59.44 | 23.33 | 12.51 | 22.56 | 15.65 | 55.13 | 27.82 | 27.82 | 20.61 | 43.2 | 26.67 | |
| MemoryOSBackbone Model=Qwen3-VL-2B-Instruct2026.05 | 71.43 | 48.21 | 78.57 | 52.59 | 13.33 | 3.39 | 25.56 | 12.58 | 50 | 30.79 | 14.29 | 14.99 | 38 | 24.98 | |
| LightMemBackbone Model=Qwen3-VL-2B-Instruct2026.05 | 55.71 | 13.93 | 16.07 | 6.89 | 66.67 | 11.8 | 13.53 | 4.78 | 38.46 | 7.74 | 26.32 | 8.15 | 30.2 | 8.08 | |
| EverMemOSBackbone Model=Qwen3-VL-2B-Instruct2026.05 | 54.29 | 12.81 | 32.14 | 14.26 | 23.33 | 11.11 | 21.05 | 7.88 | 25.64 | 7.91 | 18.05 | 10.14 | 27 | 10.19 | |
| MemOCRBackbone Model=Qwen3-VL-2B-Instruct2026.05 | 21.43 | 18.61 | 23.21 | 22.99 | 13.33 | 9.79 | 9.02 | 8.09 | 37.18 | 31.44 | 18.8 | 19.26 | 19.6 | 17.95 |