Question Answering on NarrativeQA (F1)
38.12F1 ScoreMemPro-15
Evaluation results
| Method | Links | |
|---|---|---|
| MemPro-15Backbone=GPT-4o-mini, Evolution Iterations=152026.05 | 38.12 | |
| MemPro-10Backbone=GPT-4o-mini, Evolution Iterations=102026.05 | 37.84 | |
| MemPro-5Backbone=GPT-4o-mini, Evolution Iterations=52026.05 | 37.23 | |
| GAMBackbone=GPT-4o-mini2026.05 | 36.86 | |
| SitEmb-v1.5-Qwen3 (QA)Number of retrieved chunks=top-102025.08 | 34.4 | |
| NaviRAGTokens=3305, Time (s/query)=3.232026.04 | 32.6 | |
| Qwen3 (QA)Number of retrieved chunks=top-102025.08 | 32.4 | |
| Qwen3 (out-of-box)Number of retrieved chunks=top-102025.08 | 32.2 | |
| SitEmb-v1.5-Qwen3 (QA)Number of retrieved chunks=top-52025.08 | 32 | |
| Qwen3 (QA)Number of retrieved chunks=top-52025.08 | 31.9 | |
| SitEmb-v1.5-Qwen3 (QA+SA)Number of retrieved chunks=top-102025.08 | 31.8 | |
| SitEmb-v1.5-Qwen3 (QA+SA)Number of retrieved chunks=top-52025.08 | 31.6 | |
| Full TextBackbone=GPT-4o-mini2026.05 | 31.26 | |
| MemPro-15Backbone=Qwen2.5-14B, Evolution Iterations=152026.05 | 31.23 | |
| SitEmb-v1.5-Qwen3 (QA)Number of retrieved chunks=top-32025.08 | 31.1 | |
| Qwen3 (out-of-box)Number of retrieved chunks=top-52025.08 | 30.8 | |
| MemPro-10Backbone=Qwen2.5-14B, Evolution Iterations=102026.05 | 30.78 | |
| B-FragRel-ChatGLM-C500Context Length=28K, Backbone=ChatGLM, Chunk Size=5002024.06 | 30.41 | |
| MemPro-5Backbone=Qwen2.5-14B, Evolution Iterations=52026.05 | 30.34 | |
| GraphRAGTokens=7249, Time (s/query)=14.302026.04 | 30.17 | |
| GAMBackbone=Qwen2.5-14B2026.05 | 30.17 | |
| Full TextBackbone=Qwen2.5-14B2026.05 | 29.69 | |
| B-FragRel-ChatGLM-C500Context Length=12K, Backbone=ChatGLM, Chunk Size=5002024.06 | 29.55 | |
| Qwen3 (QA)Number of retrieved chunks=top-32025.08 | 29.5 | |
| B-FragRel-ChatGLM-C800Context Length=28K, Backbone=ChatGLM, Chunk Size=8002024.06 | 29.43 | |
| SitEmb-v1.5-Qwen3 (QA+SA)Number of retrieved chunks=top-32025.08 | 29.4 | |
| Mem0Backbone=GPT-4o-mini2026.05 | 29.16 | |
| LightRAGTokens=18896, Time (s/query)=4.272026.04 | 28.98 | |
| B-FragRel-ChatGLM-C500Context Length=20K, Backbone=ChatGLM, Chunk Size=5002024.06 | 28.79 | |
| B-FragRel-ChatGLM-C800Context Length=8K, Backbone=ChatGLM, Chunk Size=8002024.06 | 28.48 | |
| B-FragRel-ChatGPT-C500Context Length=8K, Backbone=ChatGPT, Chunk Size=5002024.06 | 28.25 | |
| No-FragRel-ChatGLM-C500Context Length=12K, Backbone=ChatGLM, Chunk Size=5002024.06 | 28.2 | |
| B-FragRel-ChatGLM-C500Context Length=8K, Backbone=ChatGLM, Chunk Size=5002024.06 | 28.07 | |
| A-FragRel-ChatGLM-C500Context Length=28K, Backbone=ChatGLM, Chunk Size=5002024.06 | 28.05 | |
| B-FragRel-ChatGLM-C800Context Length=20K, Backbone=ChatGLM, Chunk Size=8002024.06 | 27.93 | |
| HippoRAG2Tokens=2635, Time (s/query)=1.362026.04 | 27.88 | |
| VanillaTokens=2560, Time (s/query)=1.012026.04 | 27.8 | |
| Mem0Backbone=Qwen2.5-14B2026.05 | 27.8 | |
| B-FragRel-ChatGLM-C800Context Length=12K, Backbone=ChatGLM, Chunk Size=8002024.06 | 27.57 | |
| A-FragRel-ChatGLM-C500Context Length=12K, Backbone=ChatGLM, Chunk Size=5002024.06 | 27.56 | |
| No-FragRel-ChatGLM-C500Context Length=28K, Backbone=ChatGLM, Chunk Size=5002024.06 | 27.51 | |
| Qwen3 (out-of-box)Number of retrieved chunks=top-32025.08 | 27.5 | |
| No-FragRel-ChatGLM-C500Context Length=8K, Backbone=ChatGLM, Chunk Size=5002024.06 | 27.44 | |
| No-FragRel-ChatGLM-C800Context Length=8K, Backbone=ChatGLM, Chunk Size=8002024.06 | 27.38 | |
| No-FragRel-ChatGLM-C800Context Length=28K, Backbone=ChatGLM, Chunk Size=8002024.06 | 27.28 | |
| A-FragRel-ChatGLM-C500Context Length=20K, Backbone=ChatGLM, Chunk Size=5002024.06 | 27.08 | |
| A-MEMBackbone=GPT-4o-mini2026.05 | 27.07 | |
| No-FragRel-ChatGLM-C800Context Length=20K, Backbone=ChatGLM, Chunk Size=8002024.06 | 26.99 | |
| No-FragRel-ChatGLM-C500Context Length=20K, Backbone=ChatGLM, Chunk Size=5002024.06 | 26.71 | |
| MemoryOSBackbone=GPT-4o-mini2026.05 | 26.7 | |
| B-FragRel-ChatGLM-C800Context Length=4K, Backbone=ChatGLM, Chunk Size=8002024.06 | 26.26 | |
| A-FragRel-ChatGLM-C500Context Length=8K, Backbone=ChatGLM, Chunk Size=5002024.06 | 26.21 | |
| No-FragRel-ChatGLM-C800Context Length=4K, Backbone=ChatGLM, Chunk Size=8002024.06 | 26.06 | |
| B-FragRel-ChatGPT-C800Context Length=8K, Backbone=ChatGPT, Chunk Size=8002024.06 | 26.04 | |
| ChatGLM3-6B-32kMemory Type=Temporary Memory2024.06 | 26 | |
| No-FragRel-ChatGLM-C800Context Length=12K, Backbone=ChatGLM, Chunk Size=8002024.06 | 25.78 | |
| CoABackbone=text-bison, Context Window=8k2024.06 | 25.26 | |
| CoABackbone=gemini-ultra, Context Window=8k2024.06 | 25.26 | |
| A-MEMBackbone=Qwen2.5-14B2026.05 | 25.18 | |
| B-FragRel-ChatGPT-C500Context Length=4K, Backbone=ChatGPT, Chunk Size=5002024.06 | 25.11 | |
| No-FragRel-ChatGPT-C500Context Length=4K, Backbone=ChatGPT, Chunk Size=5002024.06 | 25.07 | |
| RAGBackbone=GPT-4o-mini2026.05 | 25 | |
| No-FragRel-ChatGPT-C500Context Length=8K, Backbone=ChatGPT, Chunk Size=5002024.06 | 24.86 | |
| B-FragRel-ChatGPT-C800Context Length=4K, Backbone=ChatGPT, Chunk Size=8002024.06 | 24.5 | |
| B-FragRel-ChatGLM-C500Context Length=4K, Backbone=ChatGLM, Chunk Size=5002024.06 | 24.43 | |
| No-FragRel-ChatGPT-C800Context Length=8K, Backbone=ChatGPT, Chunk Size=8002024.06 | 24.06 | |
| GPT-3.5-Turbo-16kMemory Type=Temporary Memory2024.06 | 23.6 | |
| MemoryOSBackbone=Qwen2.5-14B2026.05 | 23.45 | |
| A-FragRel-ChatGLM-C500Context Length=4K, Backbone=ChatGLM, Chunk Size=5002024.06 | 22.9 | |
| No-FragRel-ChatGLM-C500Context Length=4K, Backbone=ChatGLM, Chunk Size=5002024.06 | 22.32 | |
| No-FragRel-ChatGPT-C800Context Length=4K, Backbone=ChatGPT, Chunk Size=8002024.06 | 22.25 | |
| B-FragRel-ChatGLM-C500Context Length=2K, Backbone=ChatGLM, Chunk Size=5002024.06 | 20.63 | |
| No-FragRel-ChatGLM-C500Context Length=2K, Backbone=ChatGLM, Chunk Size=5002024.06 | 20.47 | |
| CoABackbone=text-unicorn, Context Window=8k2024.06 | 20.37 | |
| B-FragRel-Llama2-C500Context Length=4K, Backbone=Llama2, Chunk Size=5002024.06 | 20.32 | |
| A-FragRel-ChatGLM-C500Context Length=2K, Backbone=ChatGLM, Chunk Size=5002024.06 | 20.21 | |
| B-FragRel-ChatGPT-C500Context Length=2K, Backbone=ChatGPT, Chunk Size=5002024.06 | 20.2 | |
| No-FragRel-ChatGPT-C500Context Length=2K, Backbone=ChatGPT, Chunk Size=5002024.06 | 20.15 | |
| B-FragRel-ChatGLM-C500Context Length=1K, Backbone=ChatGLM, Chunk Size=5002024.06 | 19.71 | |
| Vicuna-v1.5-7B-16kMemory Type=Temporary Memory2024.06 | 19.4 | |
| RAGBackbone=text-unicorn, Context Window=8k2024.06 | 19.12 | |
| No-FragRel-ChatGLM-C500Context Length=1K, Backbone=ChatGLM, Chunk Size=5002024.06 | 18.81 | |
| Llama2-7B-chat-4kMemory Type=Temporary Memory2024.06 | 18.74 | |
| A-FragRel-ChatGLM-C500Context Length=1K, Backbone=ChatGLM, Chunk Size=5002024.06 | 18.32 | |
| RAGBackbone=Qwen2.5-14B2026.05 | 18.29 | |
| No-FragRel-Llama2-C500Context Length=4K, Backbone=Llama2, Chunk Size=5002024.06 | 18.22 | |
| B-FragRel-Llama2-C500Context Length=2K, Backbone=Llama2, Chunk Size=5002024.06 | 18.02 | |
| XGen-7B-8kMemory Type=Temporary Memory2024.06 | 18 | |
| B-FragRel-ChatGPT-C500Context Length=1K, Backbone=ChatGPT, Chunk Size=5002024.06 | 17.61 | |
| LightMemBackbone=GPT-4o-mini2026.05 | 17.51 | |
| No-FragRel-Llama2-C500Context Length=2K, Backbone=Llama2, Chunk Size=5002024.06 | 17.44 | |
| B-FragRel-Llama2-C500Context Length=1K, Backbone=Llama2, Chunk Size=5002024.06 | 17.3 | |
| No-FragRel-ChatGPT-C500Context Length=1K, Backbone=ChatGPT, Chunk Size=5002024.06 | 17.19 | |
| LongChat-v1.5-7B-32kMemory Type=Temporary Memory2024.06 | 16.9 | |
| LightMemBackbone=Qwen2.5-14B2026.05 | 16.57 | |
| No-FragRel-Llama2-C500Context Length=1K, Backbone=Llama2, Chunk Size=5002024.06 | 15.75 | |
| RAGBackbone=gemini-ultra, Context Window=8k2024.06 | 14.51 | |
| VanillaBackbone=text-unicorn, Context Window=8k2024.06 | 14.45 | |
| RAGBackbone=text-bison, Context Window=8k2024.06 | 14.2 | |
| InternLM-7B-8kMemory Type=Temporary Memory2024.06 | 12.1 |