Long-context Memory Question Answering on LoCoMo (test)
33.09MultiHop BLEUTriMem
Evaluation results
| Method | Links | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| TriMemModel=Qwen3-8B, Token Cost=13392026.05 | 33.09 | 41.22 | 38.71 | 53.13 | 30.59 | 37.64 | 45.1 | 52.52 | 40.66 | 49.65 | |
| xMemoryModel=Qwen3-8B, Token Cost=22302026.05 | 28.44 | 39.13 | 28.65 | 35.41 | 17.76 | 21.57 | 40.66 | 50.57 | 34.49 | 43.51 | |
| Mem0Model=Qwen3-8B, Token Cost=11402026.05 | 28.32 | 30.07 | 23.15 | 26.15 | 11.79 | 15.15 | 30.75 | 34.97 | 27.54 | 30.1 | |
| TriMemModel=Llama-3.1-8B-Ins, Token Cost=13882026.05 | 25.42 | 34.56 | 25.98 | 32.36 | 28.4 | 32.71 | 35.76 | 43.2 | 31.37 | 38.7 | |
| SimpleMemModel=Qwen3-8B, Token Cost=6082026.05 | 23.39 | 30.39 | 24.66 | 34.51 | 14.04 | 15.39 | 35.73 | 41.26 | 29.81 | 36.25 | |
| LightMemModel=Qwen3-8B, Token Cost=7402026.05 | 22.84 | 32.54 | 37.62 | 48.37 | 18.05 | 19.02 | 23.03 | 31.37 | 25.73 | 34.36 | |
| xMemoryModel=Llama-3.1-8B-Ins, Token Cost=23752026.05 | 21.89 | 31.24 | 21.78 | 26.84 | 12.37 | 16.62 | 27.75 | 41.36 | 24.47 | 34.94 | |
| SimpleMemModel=Llama-3.1-8B-Ins, Token Cost=6742026.05 | 18.81 | 26.22 | 21.15 | 30.44 | 15.81 | 18.77 | 26.79 | 31.23 | 23.47 | 29.37 | |
| A-MemModel=Qwen3-8B, Token Cost=11802026.05 | 16.02 | 21.08 | 28.1 | 37.51 | 14.01 | 14.19 | 33.6 | 40.77 | 28.01 | 34.83 | |
| A-MemModel=Llama-3.1-8B-Ins, Token Cost=13402026.05 | 15.8 | 22.84 | 23.79 | 36.19 | 11.19 | 12.51 | 31.19 | 37.86 | 25.58 | 33.18 | |
| MemoryOSModel=Qwen3-8B, Token Cost=9112026.05 | 14.38 | 22.72 | 18.67 | 22.79 | 11.06 | 13.52 | 25.65 | 33.52 | 21.22 | 28.06 | |
| Naïve RAGModel=Llama-3.1-8B-Ins, Token Cost=11192026.05 | 14.31 | 16.18 | 7.63 | 12.28 | 9.09 | 11.58 | 22.99 | 31.02 | 17.33 | 23.18 | |
| Naïve RAGModel=Qwen3-8B, Token Cost=11192026.05 | 13.75 | 15.83 | 8.86 | 13.53 | 9.68 | 10.77 | 21.57 | 28.76 | 16.75 | 22.1 | |
| LoCoMoModel=Llama-3.1-8B-Ins, Token Cost=168632026.05 | 13.73 | 23.36 | 13.15 | 20.3 | 11.54 | 19.42 | 18.64 | 25.86 | 16.15 | 23.84 | |
| MemoryOSModel=Llama-3.1-8B-Ins, Token Cost=9642026.05 | 13.57 | 22.63 | 19.18 | 23.31 | 10.59 | 13.01 | 23.46 | 31.05 | 19.95 | 26.77 | |
| Mem0Model=Llama-3.1-8B-Ins, Token Cost=10852026.05 | 13.27 | 16.4 | 8.26 | 12.62 | 7.45 | 8.45 | 21.75 | 31.28 | 16.49 | 23.24 | |
| LightMemModel=Llama-3.1-8B-Ins, Token Cost=7582026.05 | 13.19 | 19.64 | 16.93 | 28.06 | 17.39 | 20.68 | 27.62 | 41.06 | 22.11 | 33.16 | |
| LoCoMoModel=Qwen3-8B, Token Cost=168632026.05 | 12.67 | 20.54 | 12.32 | 18.55 | 10.59 | 14.39 | 19.76 | 23.78 | 16.34 | 21.51 |