Implicit-conflict resolution on STALE 1.0 (full)
92Type I SR AccuracyGemini-3.1-pro
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| Gemini-3.1-proCategory=Closed-source LLMs2026.05 | 92 | 30 | 71 | 69 | 14 | 55 | 55.2 | |
| CUPMEMCategory=Memory Frameworks2026.05 | 91 | 78 | 32 | 89 | 75 | 43 | 68 | |
| Qwen3.5-27BCategory=Open-source LLMs2026.05 | 76 | 4 | 39 | 42 | 3.5 | 23 | 31.3 | |
| LightMemCategory=Memory Frameworks2026.05 | 52.5 | 1 | 23.5 | 21.5 | 0.5 | 7.5 | 17.8 | |
| Gemini-3.1-flash-liteCategory=Closed-source LLMs2026.05 | 41 | 1.5 | 42 | 25 | 1.5 | 23.5 | 22.4 | |
| Qwen3.5-9BCategory=Open-source LLMs2026.05 | 36 | 1 | 21.5 | 21.5 | 0 | 7.5 | 14.6 | |
| GPT-5.4Category=Closed-source LLMs2026.05 | 35 | 2 | 29 | 9 | 2 | 17 | 15.7 | |
| GPT-4o-mini*Category=Closed-source LLMs2026.05 | 30 | 0 | 11 | 9.5 | 0 | 1.5 | 8.7 | |
| GPT-5.4-nanoCategory=Closed-source LLMs2026.05 | 20.5 | 1.5 | 21.5 | 9 | 0 | 6.5 | 9.8 | |
| mem-0Category=Memory Frameworks2026.05 | 17 | 1 | 22 | 3.5 | 0 | 6.5 | 8.3 | |
| LiCoMemoryCategory=Memory Frameworks2026.05 | 15.5 | 0.5 | 22.5 | 1.5 | 1.5 | 4 | 7.6 | |
| A-memCategory=Memory Frameworks2026.05 | 13.5 | 0 | 7.5 | 8 | 0 | 1.5 | 5.1 | |
| MiniMax-M2.5Category=Open-source LLMs2026.05 | 10.5 | 1.5 | 8 | 5.5 | 5 | 2.5 | 5.5 | |
| ZepCategory=Memory Frameworks2026.05 | 10 | 0 | 19 | 3 | 1 | 3 | 6 | |
| Llama-3.3-70B-Instruct*Category=Open-source LLMs2026.05 | 6.5 | 0 | 3 | 6 | 0 | 0 | 2.6 |