Question Answering on NarrativeQA (test)
76.2ROUGE-LCriSPO
Evaluation results
| Method | Links | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| CriSPOModel=Claude Sonnet, Shots=0-shot2024.10 | 76.2 | — | — | — | — | — | — | — | — | — | |
| CriSPOLLM=Claude Sonnet, shot_count=0-shot2024.10 | 76.2 | — | — | — | — | — | — | — | — | — | |
| CriSPOModel=Claude Instant, Shots=5-shot2024.10 | 76.1 | — | — | — | — | — | — | — | — | — | |
| CriSPOLLM=Claude Instant, shot_count=5-shot2024.10 | 76.1 | — | — | — | — | — | — | — | — | — | |
| CriSPOModel=Claude Sonnet, Shots=5-shot2024.10 | 75.2 | — | — | — | — | — | — | — | — | — | |
| CriSPOLLM=Claude Sonnet, shot_count=5-shot2024.10 | 75.2 | — | — | — | — | — | — | — | — | — | |
| CriSPOModel=Claude Instant, Shots=0-shot2024.10 | 75.1 | — | — | — | — | — | — | — | — | — | |
| CriSPOLLM=Claude Instant, shot_count=0-shot2024.10 | 75.1 | — | — | — | — | — | — | — | — | — | |
| ManualModel=Claude Instant, Shots=5-shot2024.10 | 67 | — | — | — | — | — | — | — | — | — | |
| ManualLLM=Claude Instant, shot_count=5-shot2024.10 | 67 | — | — | — | — | — | — | — | — | — | |
| ManualModel=Claude Sonnet, Shots=5-shot2024.10 | 66.7 | — | — | — | — | — | — | — | — | — | |
| ManualLLM=Claude Sonnet, shot_count=5-shot2024.10 | 66.7 | — | — | — | — | — | — | — | — | — | |
| ManualModel=Claude Instant, Shots=0-shot2024.10 | 64.2 | — | — | — | — | — | — | — | — | — | |
| ManualLLM=Claude Instant, shot_count=0-shot2024.10 | 64.2 | — | — | — | — | — | — | — | — | — | |
| ManualModel=Claude Sonnet, Shots=0-shot2024.10 | 64 | — | — | — | — | — | — | — | — | — | |
| ManualLLM=Claude Sonnet, shot_count=0-shot2024.10 | 64 | — | — | — | — | — | — | — | — | — | |
| MemGenBackbone=Qwen3-8B2026.01 | 63.94 | — | — | — | — | — | — | — | 50.88 | — | |
| Reasoner-onlyBackbone=Qwen3-8B2026.01 | 63.81 | — | — | — | — | — | — | — | 51 | — | |
| PECAN2024.10 | 60.2 | — | — | — | — | 61.1 | 31 | 1 | — | — | |
| OPROModel=Claude Instant2024.10 | 59.4 | — | — | — | — | — | — | — | — | — | |
| OPROLLM=Claude Instant2024.10 | 59.4 | — | — | — | — | — | — | — | — | — | |
| OPROModel=Claude Sonnet2024.10 | 58.6 | — | — | — | — | — | — | — | — | — | |
| OPROLLM=Claude Sonnet2024.10 | 58.6 | — | — | — | — | — | — | — | — | — | |
| A-MemBackbone=SmolLM3-3B2026.01 | 55.99 | — | — | — | — | — | — | — | 19.88 | — | |
| In-context (Oracle)Backbone=QWEN2.5-7B, Setting=Oracle access to ground-truth context2026.05 | 55.03 | — | — | — | — | — | — | — | — | 55.99 | |
| DragonTop-k=72024.10 | 54.2 | — | — | — | — | 55.1 | 37.5 | 1.21 | — | — | |
| MemGenBackbone=Qwen2.5-1.5B-Instruct2026.01 | 54.2 | — | — | — | — | — | — | — | 42.63 | — | |
| DragonTop-k=52024.10 | 52.9 | — | — | — | — | 53.8 | 26.9 | 0.87 | — | — | |
| BM25Top-k=72024.10 | 52.9 | — | — | — | — | 53.7 | 37.5 | 1.21 | — | — | |
| Llama-3.1-8B2024.10 | 52.6 | — | — | — | — | 53.7 | 3,361.9 | 108.45 | — | — | |
| BM25Top-k=52024.10 | 51.8 | — | — | — | — | 52.7 | 26.7 | 0.86 | — | — | |
| RAPTOR-CTStrategy=collapsed tree, Top-k=202024.10 | 51.2 | — | — | — | — | 52 | 35.1 | 1.13 | — | — | |
| A-MemBackbone=Qwen2.5-1.5B-Instruct2026.01 | 49.85 | — | — | — | — | — | — | — | 15.62 | — | |
| LongLLMLingua2024.10 | 49.5 | — | — | — | — | 50.5 | 1,789.4 | 57.72 | — | — | |
| RAPTOR-CTStrategy=collapsed tree, Top-k=52024.10 | 47.8 | — | — | — | — | 48.6 | 17.9 | 0.58 | — | — | |
| MemGenBackbone=SmolLM3-3B2026.01 | 47.37 | — | — | — | — | — | — | — | 49.38 | — | |
| RAGBackbone=Qwen3-8B2026.01 | 47.29 | — | — | — | — | — | — | — | 14.87 | — | |
| RAGBackbone=QWEN2.5-7B, k=32026.05 | 46.61 | — | — | — | — | — | — | — | — | 47.36 | |
| A-MemBackbone=Qwen3-8B2026.01 | 46.04 | — | — | — | — | — | — | — | 15.88 | — | |
| RAGBackbone=SmolLM3-3B2026.01 | 40.12 | — | — | — | — | — | — | — | 6.12 | — | |
| RAPTOR-TTStrategy=tree traversal2024.10 | 39.8 | — | — | — | — | 40.6 | 20.3 | 0.65 | — | — | |
| In-context (Oracle)Backbone=QWEN2.5-0.5B, Setting=Oracle access to ground-truth context2026.05 | 39.64 | — | — | — | — | — | — | — | — | 40.65 | |
| SBERTTop-k=72024.10 | 38.8 | — | — | — | — | 39.5 | 37.5 | 1.21 | — | — | |
| LATENTGRAPHMEMBackbone=Qwen3-8B2026.01 | 37.6 | — | — | — | — | — | — | — | 51 | — | |
| Reasoner-onlyBackbone=SmolLM3-3B2026.01 | 37.43 | — | — | — | — | — | — | — | 40.5 | — | |
| Ours (Oracle Setting)Backbone=QWEN2.5-7B, Setting=Oracle access to adapter2026.05 | 36.78 | — | — | — | — | — | — | — | — | 37.55 | |
| SBERTTop-k=52024.10 | 35.8 | — | — | — | — | 36.5 | 26.8 | 0.86 | — | — | |
| LATENTGRAPHMEMBackbone=SmolLM3-3B2026.01 | 35.38 | — | — | — | — | — | — | — | 47 | — | |
| RAGBackbone=QWEN2.5-7B, k=52026.05 | 34.87 | — | — | — | — | — | — | — | — | 35.57 | |
| LATENTGRAPHMEMBackbone=Qwen2.5-1.5B-Instruct2026.01 | 32.25 | — | — | — | — | — | — | — | 45.5 | — | |
| Reader-to-Retriever DistillationInitialization=BM25, Iteration=12020.12 | 32 | 35.3 | 7.5 | 11.1 | — | — | — | — | — | — | |
| Retriever + Reader2026.04 | 32 | 35.3 | 7.5 | 11.1 | — | — | — | — | — | — | |
| DPR + FiD2020.12 | 30.8 | 34 | 6.9 | 10.8 | — | — | — | — | — | — | |
| Reader-to-Retriever DistillationInitialization=BM25, Iteration=02020.12 | 30.3 | 33.7 | 6.5 | 10.4 | — | — | — | — | — | — | |
| Ours (Oracle Setting)Backbone=QWEN2.5-0.5B, Setting=Oracle access to adapter2026.05 | 30.22 | — | — | — | — | — | — | — | — | 31.23 | |
| Reasoner-onlyBackbone=Qwen2.5-1.5B-Instruct2026.01 | 28.13 | — | — | — | — | — | — | — | 41.5 | — | |
| Ours (Retrieval)Backbone=QWEN2.5-7B, k=52026.05 | 27.86 | — | — | — | — | — | — | — | — | 28.64 | |
| Ours (Retrieval)Backbone=QWEN2.5-7B, k=32026.05 | 27.68 | — | — | — | — | — | — | — | — | 28.46 | |
| DPRLanguage Model=GPT-4o-mini2026.04 | 26.5 | 21.5 | 1.3 | 14.2 | — | — | — | — | — | — | |
| RAGBackbone=Qwen2.5-1.5B-Instruct2026.01 | 26.47 | — | — | — | — | — | — | — | 4.5 | — | |
| DTCRSLanguage Model=GPT-4o-mini2026.04 | 26 | 21.1 | 1.3 | 14.4 | — | — | — | — | — | — | |
| RAPTORLanguage Model=GPT-4o-mini2026.04 | 25 | 21.2 | 1.1 | 14.1 | — | — | — | — | — | — | |
| Ours (Retrieval)Backbone=QWEN2.5-0.5B, k=32026.05 | 23.57 | — | — | — | — | — | — | — | — | 24.54 | |
| Ours (Retrieval)Backbone=QWEN2.5-0.5B, k=52026.05 | 23.46 | — | — | — | — | — | — | — | — | 24.44 | |
| Recursively Summarizing Books2026.04 | 21.6 | 22.3 | 4.2 | 10.6 | — | — | — | — | — | — | |
| RAGBackbone=QWEN2.5-0.5B, k=32026.05 | 20.21 | — | — | — | — | — | — | — | — | 21.12 | |
| RAGBackbone=QWEN2.5-0.5B, k=52026.05 | 15.79 | — | — | — | — | — | — | — | — | 16.66 | |
| BM25 + BERT2026.04 | 15.5 | 14.5 | 1.4 | 5 | — | — | — | — | — | — | |
| Base ModelBackbone=QWEN2.5-7B2026.05 | 14.4 | — | — | — | — | — | — | — | — | 14.95 | |
| Kočiskỳ et al.2020.12 | 14 | 19.1 | 2.1 | 4.4 | — | — | — | — | — | — | |
| Base ModelBackbone=QWEN2.5-0.5B2026.05 | 12.26 | — | — | — | — | — | — | — | — | 13.08 | |
| PREMemBackbone=Qwen2.5-1.5B-Instruct2026.01 | 11.64 | — | — | — | — | — | — | — | 10.37 | — | |
| MeMWalker2024.10 | 9.8 | — | — | — | — | 11.2 | 353.8 | 11.41 | — | — | |
| InfiniteICLBackbone=QWEN2.5-0.5B2026.05 | 9.35 | — | — | — | — | — | — | — | — | 10.1 | |
| PREMemBackbone=SmolLM3-3B2026.01 | 9.11 | — | — | — | — | — | — | — | 11.22 | — | |
| THEANINEBackbone=SmolLM3-3B2026.01 | 7.84 | — | — | — | — | — | — | — | 10.87 | — | |
| BiDAF2026.04 | 6.2 | 5.7 | 0.3 | 3.7 | — | — | — | — | — | — | |
| TempLoraCDBackbone=QWEN2.5-0.5B2026.05 | 6.14 | — | — | — | — | — | — | — | — | 6.65 | |
| Mem0Backbone=Qwen2.5-1.5B-Instruct2026.01 | 5.23 | — | — | — | — | — | — | — | 1.12 | — | |
| THEANINEBackbone=Qwen3-8B2026.01 | 5.13 | — | — | — | — | — | — | — | 6.63 | — | |
| TempLoraBackbone=QWEN2.5-0.5B2026.05 | 4.18 | — | — | — | — | — | — | — | — | 4.67 | |
| TempLoraBackbone=QWEN2.5-7B2026.05 | 4.11 | — | — | — | — | — | — | — | — | 4.83 | |
| THEANINEBackbone=Qwen2.5-1.5B-Instruct2026.01 | 3.84 | — | — | — | — | — | — | — | 2.25 | — | |
| TempLoraCDBackbone=QWEN2.5-7B2026.05 | 3.78 | — | — | — | — | — | — | — | — | 4.09 | |
| InfiniteICLBackbone=QWEN2.5-7B2026.05 | 3.77 | — | — | — | — | — | — | — | — | 4.24 | |
| Mem0Backbone=Qwen3-8B2026.01 | 3.12 | — | — | — | — | — | — | — | 0.5 | — | |
| Mem0Backbone=SmolLM3-3B2026.01 | 3.04 | — | — | — | — | — | — | — | 0.25 | — | |
| PREMemBackbone=Qwen3-8B2026.01 | 3 | — | — | — | — | — | — | — | 2.57 | — | |
| AutoComCompression Ratio=4x2026.03 | — | — | — | — | — | 14.41 | — | — | — | — | |
| BART-LS2022.09 | — | — | — | — | 26.2 | — | — | — | — | — | |
| BM25Executive Model=Qwen2.5-32B-Instruct2026.05 | — | — | — | — | — | — | — | — | 10.24 | — | |
| BM25Executive Model=Gemini-3-Flash2026.05 | — | — | — | — | — | — | — | — | 14.33 | — | |
| CartridgesExecutive Model=Qwen2.5-32B-Instruct2026.05 | — | — | — | — | — | — | — | — | 3.75 | — | |
| HippoRAG2Executive Model=Qwen2.5-32B-Instruct2026.05 | — | — | — | — | — | — | — | — | 21.39 | — | |
| HippoRAG2Executive Model=Gemini-3-Flash2026.05 | — | — | — | — | — | — | — | — | 23.21 | — | |
| ICAECompression Ratio=4x2026.03 | — | — | — | — | — | 15.76 | — | — | — | — | |
| LLOCOCompression Ratio=4x2026.03 | — | — | — | — | — | 15.88 | — | — | — | — | |
| LongT5-3BModel size=3B2022.09 | — | — | — | — | 29.3 | — | — | — | — | — | |
| LongT5-baseModel size=base2022.09 | — | — | — | — | 23 | — | — | — | — | — | |
| LongT5-largeModel size=large2022.09 | — | — | — | — | 27.2 | — | — | — | — | — |