Memory-augmented language modeling evaluation on LONGMEMEVAL-S
73.2AccuracyNaiveRAG
Evaluation Results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| NaiveRAGBackbone=GLM-4.62025.10 | 73.2 | — | — | — | — | — | — | 53,725.15 | |
| LightMemBackbone=GLM-4.6, r (compression rate)=0.6, th (capacity threshold)=256, Update protocol=Online2025.10 | 73.2 | 16.55 | 13.99 | — | — | 30.54 | 10.78 | 1,077.69 | |
| LightMemBackbone=GLM-4.6, r (compression rate)=0.5, th (capacity threshold)=256, Update protocol=Online2025.10 | 73 | 16.55 | 14.06 | — | — | 30.61 | 10.78 | 1,014.37 | |
| LightMemBackbone=GLM-4.6, r (compression rate)=0.7, th (capacity threshold)=512, Update protocol=Online2025.10 | 72.8 | 16.55 | 13.91 | — | — | 30.46 | 10.78 | 1,038.19 | |
| A-MEMBackbone=GLM-4.62025.10 | 70.6 | 444.95 | 63.4 | 1,992.04 | 403.39 | 2,903.78 | 450.4 | 8,068.8 | |
| LightMemBackbone=Qwen3-30B-A3B-Instruct-2507, r (compression rate)=0.6, th (capacity threshold)=768, Update protocol=Online2025.10 | 70.2 | 13.19 | 19.21 | — | — | 32.4 | 19.97 | 417.13 | |
| LightMemBackbone=Qwen3-30B-A3B-Instruct-2507, r (compression rate)=0.8, th (capacity threshold)=1024, Update protocol=Online2025.10 | 68.69 | 14.82 | 18.49 | — | — | 33.31 | 9.43 | 355.71 | |
| LightMemBackbone=GPT-4o-mini, r (compression rate)=0.7, th (capacity threshold)=512, Update protocol=Online2025.10 | 68.64 | 18.88 | 9.37 | — | — | 28.25 | 18.43 | 283.76 | |
| LightMemBackbone=GPT-4o-mini, r (compression rate)=0.6, th (capacity threshold)=256, Update protocol=Online2025.10 | 67.78 | 24.58 | 10.53 | — | — | 35.11 | 30.47 | 329.61 | |
| LightMemBackbone=Qwen3-30B-A3B-Instruct-2507, r (compression rate)=0.8, th (capacity threshold)=1024, Update protocol=OP-update2025.10 | 67.34 | — | — | 106.91 | 6.2 | 113.11 | 168.37 | 1,026.9 | |
| LightMemBackbone=GPT-4o-mini, r (compression rate)=0.7, th (capacity threshold)=512, Update protocol=OP-update2025.10 | 67.07 | — | — | 79.38 | 4.06 | 83.44 | 125.47 | 496.03 | |
| LightMemBackbone=GPT-4o-mini, r (compression rate)=0.6, th (capacity threshold)=256, Update protocol=OP-update2025.10 | 65.39 | — | — | 53.98 | 3.18 | 57.16 | 85.07 | 411.56 | |
| A-MEMBackbone=Qwen3-30B-A3B-Instruct-25072025.10 | 65.2 | 219.21 | 66.98 | 1,260.54 | 318.2 | 1,864.93 | 989.3 | 5,367.51 | |
| LightMemBackbone=Qwen3-30B-A3B-Instruct-2507, r (compression rate)=0.6, th (capacity threshold)=768, Update protocol=OP-update2025.10 | 65.14 | — | — | 97.11 | 5.92 | 103.03 | 152.93 | 1,023.56 | |
| LightMemBackbone=GPT-4o-mini, r (compression rate)=0.5, th (capacity threshold)=256, Update protocol=OP-update2025.10 | 64.69 | — | — | 44.46 | 2.56 | 47.02 | 70.23 | 342.63 | |
| LightMemBackbone=GPT-4o-mini, r (compression rate)=0.5, th (capacity threshold)=256, Update protocol=Online2025.10 | 64.29 | 20.8 | 10.01 | — | — | 30.81 | 25.67 | 302.69 | |
| A-MEMBackbone=GPT-4o-mini2025.10 | 62.6 | 214.66 | 42.82 | 1,157.52 | 190.81 | 1,605.81 | 986.55 | 5,132.06 | |
| LightMemBackbone=Qwen3-30B-A3B-Instruct-2507, r (compression rate)=0.4, th (capacity threshold)=768, Update protocol=OP-update2025.10 | 62.34 | — | — | 111.13 | 7.88 | 119.01 | 176.02 | 1,036.47 | |
| LightMemBackbone=Qwen3-30B-A3B-Instruct-2507, r (compression rate)=0.4, th (capacity threshold)=768, Update protocol=Online2025.10 | 61.95 | 9.01 | 16.14 | — | — | 25.15 | 16.54 | 357.13 | |
| NaiveRAGBackbone=GPT-4o-mini2025.10 | 61 | — | — | — | — | — | — | 867.38 | |
| NaiveRAGBackbone=Qwen3-30B-A3B-Instruct-25072025.10 | 60.8 | — | — | — | — | — | — | 659.09 | |
| FullTextBackbone=GPT-4o-mini2025.10 | 56.8 | — | — | — | — | 105.07 | — | — | |
| FullTextBackbone=Qwen3-30B-A3B-Instruct-25072025.10 | 54.8 | — | — | — | — | 105.07 | — | — | |
| Mem0Backbone=GPT-4o-mini2025.10 | 53.61 | 424.13 | 17.76 | 560.17 | 150.56 | 1,152.62 | 811.57 | 4,248.49 | |
| LangMemBackbone=Qwen3-30B-A3B-Instruct-25072025.10 | 50.8 | — | — | 1,311.96 | 118.06 | 1,430.02 | 495.12 | 3,237.16 | |
| MemoryOSBackbone=Qwen3-30B-A3B-Instruct-25072025.10 | 49.6 | 2,101.54 | 510.88 | 305.12 | 27.43 | 2,944.97 | 2,922.28 | 8,721.78 | |
| LangMemBackbone=GLM-4.62025.10 | 49.2 | — | — | 3,052.42 | 7.03 | 3,059.45 | 314.61 | 5,577.91 | |
| MemoryOSBackbone=GPT-4o-mini2025.10 | 44.8 | 2,302.35 | 304.18 | 350.02 | 35.19 | 2,991.75 | 2,938.41 | 8,030.04 | |
| Mem0Backbone=Qwen3-30B-A3B-Instruct-25072025.10 | 39.51 | 424.2 | 15.34 | 411.5 | 111.35 | 1,001.9 | 722.76 | 2,239.94 | |
| LangMemBackbone=GPT-4o-mini2025.10 | 37.2 | — | — | 982.68 | 119.48 | 1,102.16 | 520.62 | 2,293.7 | |
| FullTextBackbone=GLM-4.62025.10 | 36.71 | — | — | — | — | 103.38 | — | — |