Memory Capability Evaluation on BEAM 100K tokens
5Abstention RateMemoryOS
Evaluation results
| Method | Links | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| MemoryOSJudge LLM=GPT-4.1-mini2026.05 | 5 | 10.5 | 12.7 | 36.4 | 52.6 | 40.9 | 25.2 | 54.5 | 22.4 | 26.2 | 28.64 | |
| LightMemJudge LLM=GPT-4.1-mini2026.05 | 10 | 15.4 | 18.1 | 50.8 | 45.6 | 41.6 | 30 | 59.8 | 28.5 | 24.8 | 32.46 | |
| SimpleMemJudge LLM=GPT-4.1-mini2026.05 | 12.5 | 20.6 | 25.1 | 60.7 | 60 | 55 | 38.1 | 68.1 | 30.7 | 35 | 40.68 | |
| Llama Maverick (Vanilla)Model=Llama Maverick, Setup=Vanilla2025.10 | 20 | 2.5 | 19 | 51 | 41.2 | 30 | 15.2 | 45 | 6.5 | 10 | 24 | |
| Mem0Judge LLM=GPT-4.1-mini2026.05 | 20 | 18.4 | 22.6 | 58.4 | 57.8 | 45.1 | 33.3 | 52.5 | 30.9 | 31.5 | 37.05 | |
| NEMORIJudge LLM=GPT-4.1-mini2026.05 | 25 | 18.9 | 20.4 | 63.5 | 64.5 | 52.1 | 32.4 | 64 | 31.7 | 32 | 40.45 | |
| Qwen 2.5 (Vanilla)Model=Qwen 2.5, Setup=Vanilla2025.10 | 30 | 3.1 | 19.2 | 42.5 | 40 | 43.7 | 22.2 | 55.4 | 12.8 | 11.2 | 28 | |
| MEMIRJudge LLM=GPT-4.1-mini2026.05 | 37.5 | 32.3 | 28.6 | 69.8 | 65.6 | 58.4 | 43.7 | 71.4 | 36.8 | 38.5 | 48.26 | |
| LIGHTModel=Qwen 2.5, Setup=LIGHT2025.10 | 47.5 | 3.7 | 20.5 | 47.9 | 36.2 | 36.2 | 28.1 | 56.6 | 23.2 | 11.2 | 31.1 | |
| GPT-4.1-nano (Vanilla)Model=GPT-4.1-nano, Setup=Vanilla2025.10 | 47.5 | 1.2 | 18.1 | 27.3 | 42.5 | 27.5 | 17.8 | 43.7 | 2.8 | 11.2 | 23.9 | |
| LIGHTModel=GPT-4.1-nano, Setup=LIGHT2025.10 | 57.5 | 3.1 | 17.7 | 53.8 | 40 | 37.5 | 36.5 | 62.5 | 20.2 | 16.2 | 34.5 | |
| LIGHTModel=Llama Maverick, Setup=LIGHT2025.10 | 60 | 3.1 | 16.6 | 51.8 | 41.2 | 45 | 35.3 | 62.5 | 23.8 | 18.7 | 35.8 | |
| Qwen 2.5 (RAG)Model=Qwen 2.5, Setup=RAG2025.10 | 65 | 2.5 | 20.1 | 33.8 | 37.5 | 27.5 | 20.3 | 37.9 | 7.4 | 16.2 | 26.9 | |
| LIGHTModel=Gemini 2 Flash, Setup=LIGHT2025.10 | 67.5 | 1.8 | 16.6 | 46.4 | 36.2 | 30 | 22.5 | 46.2 | 13.9 | 12.5 | 29.4 | |
| Llama Maverick (RAG)Model=Llama Maverick, Setup=RAG2025.10 | 80 | 3.1 | 16.2 | 39.2 | 37.5 | 35 | 22.5 | 51.2 | 11.1 | 27.5 | 32.3 | |
| Gemini 2 Flash (Vanilla)Model=Gemini 2 Flash, Setup=Vanilla2025.10 | 80 | 0.6 | 18.1 | 33.3 | 27.5 | 12.5 | 20 | 30 | 1.8 | 18.7 | 24.2 | |
| Gemini 2 Flash (RAG)Model=Gemini 2 Flash, Setup=RAG2025.10 | 80 | 5 | 19.1 | 34.1 | 28.7 | 32.5 | 14.8 | 41.6 | 9.3 | 15 | 28 | |
| GPT-4.1-nano (RAG)Model=GPT-4.1-nano, Setup=RAG2025.10 | 80 | 1.8 | 16.9 | 36.2 | 35 | 37.5 | 26.3 | 55 | 8.3 | 12.5 | 30.9 |