Long-context Question Answering on LongBench (test)
7,011HotpotQALingoEDU
Evaluation results
| Method | Links | ||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| LingoEDUBackbone=GPT-4.1, Approach=EDU-based context compression2025.12 | 7,011 | — | — | — | 7,468 | — | 5,486 | — | — | — | — | — | — | — | — | — | — | — | 25.34 | — | — | — | — | — | — | — | |
| Self-SumBackbone=GPT-4.1, Approach=LLM-generated abstractive summary2025.12 | 6,789 | — | — | — | 7,439 | — | 5,348 | — | — | — | — | — | — | — | — | — | — | — | 23.51 | — | — | — | — | — | — | — | |
| GlyphApproach=Implicit compression2025.12 | 6,642 | — | — | — | 7,298 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| StandardBackbone=GPT-4.1, Approach=Full context2025.12 | 6,583 | — | — | — | 7,298 | — | 5,190 | — | — | — | — | — | — | — | — | — | — | — | 21.8 | — | — | — | — | — | — | — | |
| LingoEDUBackbone=Gemini-2.5-Pro, Approach=EDU-based context compression2025.12 | 4,046 | — | — | — | 4,091 | — | 3,122 | — | — | — | — | — | — | — | — | — | — | — | 8.12 | — | — | — | — | — | — | — | |
| Self-SumBackbone=Gemini-2.5-Pro, Approach=LLM-generated abstractive summary2025.12 | 3,778 | — | — | — | 3,990 | — | 3,077 | — | — | — | — | — | — | — | — | — | — | — | 7.79 | — | — | — | — | — | — | — | |
| StandardBackbone=Gemini-2.5-Pro, Approach=Full context2025.12 | 3,520 | — | — | — | 3,810 | — | 2,855 | — | — | — | — | — | — | — | — | — | — | — | 7.15 | — | — | — | — | — | — | — | |
| InfMem +RLModel=Qwen3-4B2026.02 | 65.14 | — | 20.77 | — | 74.76 | 40.74 | 53.22 | 50.93 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MemAgent +RLModel=Qwen3-4B2026.02 | 63.8 | — | 20.74 | — | 67.83 | 41.02 | 42.14 | 47.11 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| InfMemModel=Qwen3-4B2026.02 | 60.96 | — | 23.27 | — | 69.66 | 35.14 | 44.19 | 46.64 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| InfMem +RLModel=Qwen2.5-7B2026.02 | 60.34 | — | 20.43 | — | 65.19 | 35.68 | 50.66 | 46.46 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| InfMem +RLModel=Qwen3-1.7B2026.02 | 59.28 | — | 19.23 | — | 55.02 | 33.19 | 40.98 | 41.54 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MemAgentModel=Qwen3-4B2026.02 | 57.67 | — | 20.22 | — | 59.09 | 33.52 | 32.12 | 40.52 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MemAgent +RLModel=Qwen2.5-7B2026.02 | 56.17 | — | 19.47 | — | 57.66 | 35.52 | 31.23 | 40.01 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MemAgentModel=Qwen2.5-7B2026.02 | 53.23 | — | 19.86 | — | 55.4 | 31.63 | 36.52 | 39.33 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| YaRNModel=Qwen3-4B2026.02 | 53.2 | — | 21.46 | — | 50.31 | 40.14 | 32.18 | 39.46 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| InfMemModel=Qwen2.5-7B2026.02 | 52.95 | — | 19.76 | — | 48.78 | 31.09 | 31.69 | 36.85 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ECoRAGReader Model=Llama3-8B2025.06 | 52.29 | — | — | — | — | — | 24.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MemAgent +RLModel=Qwen3-1.7B2026.02 | 50.22 | — | 19.23 | — | 47.58 | 35.48 | 30.35 | 35.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| standard RAGReader Model=Llama3-8B2025.06 | 49.76 | — | — | — | — | — | 23.91 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| RECOMPReader Model=Llama3-8B2025.06 | 49.12 | — | — | — | — | — | 23.08 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| InfMemModel=Qwen3-1.7B2026.02 | 48.73 | — | 20.25 | — | 54.05 | 33.91 | 28.4 | 37.07 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| YaRNModel=Qwen2.5-7B2026.02 | 42.92 | — | 16.12 | — | 40.55 | 28.84 | 19.28 | 29.54 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ARKCategory=Proposed Framework, Base Model=Qwen3-embedding2025.11 | 42.35 | — | 21.57 | — | 23.41 | 24.04 | 20.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MemAgentModel=Qwen3-1.7B2026.02 | 41.68 | — | 15.04 | — | 34.04 | 30.94 | 19.84 | 28.31 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Jina-emb-v3Category=Dense Retrieval model2025.11 | 39.17 | — | 19.39 | — | 19.34 | 20.7 | 20.58 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 63.08 | 54.84 | 54.9 | 57.41 | 43.24 | |
| BGE-M3Category=Dense Retrieval model2025.11 | 38.64 | — | 18.37 | — | 22.86 | 23.33 | 21.13 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 60 | 50.77 | 56 | 52.54 | 52.83 | |
| Stella-v5Category=Dense Retrieval model2025.11 | 35.45 | — | 20.9 | — | 22.13 | 23.39 | 17.08 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 65.67 | 66.67 | 58 | 50 | 67.39 | |
| Qwen3-embeddingCategory=Dense Retrieval model, Variant=0.6B2025.11 | 35.27 | — | 19.58 | — | 21.24 | 23.9 | 14.19 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 58.33 | 52.54 | 63.46 | 57.14 | 68.89 | |
| YaRNModel=Qwen3-1.7B2026.02 | 33.87 | — | 17.09 | — | 50.32 | 37.91 | 23.86 | 32.61 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PyramidKVRatio=16.8%, Base Model=LLaMA2-7b-chat2025.07 | 31.22 | — | 16.95 | — | 25.5 | 20.81 | 36.05 | 32.47 | 20.77 | 20.52 | 22.53 | 64 | 83.72 | 40.09 | 6 | 10 | 58.36 | 53.33 | — | — | — | — | — | — | — | — | |
| SpindleKVRatio=26.0%, Base Model=LLaMA2-7b-chat2025.07 | 31.09 | — | 17.31 | — | 25.8 | 20.72 | 36.86 | 33.1 | 22.93 | 21.32 | 23.94 | 64 | 84 | 40.97 | 6 | 10 | 60.92 | 53.7 | — | — | — | — | — | — | — | — | |
| SpindleKVRatio=16.7%, Base Model=LLaMA2-7b-chat2025.07 | 31.01 | — | 17.33 | — | 25.88 | 20.64 | 35.04 | 32.86 | 21.28 | 20.15 | 22.67 | 64 | 83.31 | 39.81 | 5.5 | 11 | 59.6 | 54.34 | — | — | — | — | — | — | — | — | |
| PyramidInferRatio=21.10%, Base Model=LLaMA2-7b-chat2025.07 | 30.97 | — | 13.39 | — | 27.24 | 14.44 | 13.23 | 26.69 | 19.67 | 18.71 | 20.49 | 43.5 | 70.34 | 37.94 | 3 | 2 | 52.44 | 51.41 | — | — | — | — | — | — | — | — | |
| FullKVRatio=100%, Base Model=LLaMA2-7b-chat2025.07 | 30.92 | — | 18.39 | — | 25.73 | 20.14 | 35.67 | 33.39 | 25.58 | 20.98 | 26.43 | 64 | 83.38 | 41.02 | 5.5 | 10 | 60.81 | 55.12 | — | — | — | — | — | — | — | — | |
| PyramidKVRatio=26.3%, Base Model=LLaMA2-7b-chat2025.07 | 30.86 | — | 17.32 | — | 25.62 | 20.99 | 36.37 | 32.91 | 22.38 | 20.57 | 23.24 | 64 | 83.81 | 40.82 | 6 | 10.5 | 59.95 | 54.4 | — | — | — | — | — | — | — | — | |
| SpindleKVRatio=41.1%, Base Model=LLaMA2-7b-chat2025.07 | 30.8 | — | 18.45 | — | 25.74 | 21.23 | 36.67 | 33.41 | 24.49 | 20.66 | 25.18 | 64 | 84.31 | 41.11 | 6 | 10 | 60.45 | 54.92 | — | — | — | — | — | — | — | — | |
| PyramidKVRatio=22.3%, Base Model=LLaMA2-7b-chat2025.07 | 30.78 | — | 17.77 | — | 25.96 | 21.77 | 35.7 | 32.83 | 21.74 | 20.5 | 23.48 | 64 | 83.8 | 40.33 | 6 | 10 | 59.69 | 53.83 | — | — | — | — | — | — | — | — | |
| SpindleKVRatio=30.8%, Base Model=LLaMA2-7b-chat2025.07 | 30.77 | — | 18.13 | — | 26.1 | 19.95 | 36.91 | 33 | 23.19 | 20.95 | 24.35 | 64 | 83.69 | 41.37 | 6 | 9 | 59.85 | 53.77 | — | — | — | — | — | — | — | — | |
| PyramidKVRatio=41.3%, Base Model=LLaMA2-7b-chat2025.07 | 30.76 | — | 18.38 | — | 25.45 | 20.99 | 35.98 | 33.31 | 23.73 | 20.88 | 25.08 | 64 | 83.75 | 41.17 | 6 | 10.5 | 60.58 | 54.93 | — | — | — | — | — | — | — | — | |
| SpindleKVRatio=22.1%, Base Model=LLaMA2-7b-chat2025.07 | 30.69 | — | 17.34 | — | 25.64 | 21.2 | 35.81 | 32.89 | 22.31 | 20.64 | 23.17 | 64 | 84.15 | 40.65 | 6 | 10 | 60.31 | 54.49 | — | — | — | — | — | — | — | — | |
| PyramidKVRatio=30.8%, Base Model=LLaMA2-7b-chat2025.07 | 30.55 | — | 17.78 | — | 26.04 | 20.49 | 36.86 | 33.13 | 22.92 | 20.97 | 24.13 | 64 | 83.59 | 41.06 | 6 | 11 | 60.74 | 54.15 | — | — | — | — | — | — | — | — | |
| PyramidInferRatio=40.7%, Base Model=LLaMA2-7b-chat2025.07 | 29.14 | — | 15.36 | — | 24.53 | 15.4 | 19.23 | 29.09 | 21.64 | 19.66 | 22.7 | 54 | 81.79 | 40.71 | 4 | 3.5 | 54.29 | 51.98 | — | — | — | — | — | — | — | — | |
| HippoRAGCategory=Advanced RAG method2025.11 | 28.71 | — | 11.51 | — | 30.96 | 21.9 | 13.09 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 87.34 | 53.85 | 58.21 | 34.67 | 60.76 | |
| Qwen3-0.6B-ALLMEMArchitecture=ALLMEM, Avg. Len.=370522026.02 | 28.23 | — | 12.31 | — | — | — | 10.76 | 27.33 | — | — | — | — | — | — | — | — | — | — | 25.71 | 16.32 | 70.64 | — | — | — | — | — | |
| PyramidInferRatio=31.2%, Base Model=LLaMA2-7b-chat2025.07 | 27.69 | — | 13.8 | — | 26.1 | 15.27 | 17.69 | 28.28 | 20.53 | 19.42 | 22.04 | 53.5 | 77.06 | 40.5 | 2 | 6.5 | 52.69 | 50.46 | — | — | — | — | — | — | — | — | |
| Qwen3-0.6BAttention=Full attention, Avg. Len.=370522026.02 | 27.21 | — | 13.38 | — | — | — | 10.86 | 28.16 | — | — | — | — | — | — | — | — | — | — | 26.03 | 16.86 | 74.62 | — | — | — | — | — | |
| PyramidInferRatio=15.9%, Base Model=LLaMA2-7b-chat2025.07 | 26.98 | — | 12.11 | — | 27.3 | 14.49 | 14.25 | 25.56 | 19.28 | 18.33 | 19.78 | 38 | 61.76 | 38.84 | 2 | 6 | 52.35 | 50.78 | — | — | — | — | — | — | — | — | |
| FullDescription=Directly providing the entire context to an LLM2025.11 | 26.87 | — | 12.95 | — | 20.13 | 22.79 | 6.74 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 83.33 | 46.03 | 80 | 64.52 | 70.97 | |
| PyramidInferRatio=25.9%, Base Model=LLaMA2-7b-chat2025.07 | 26.82 | — | 14.85 | — | 24.95 | 15.19 | 15.41 | 27.5 | 19.96 | 18.66 | 21.34 | 49 | 76.22 | 38.83 | 4.5 | 5 | 51.87 | 51.71 | — | — | — | — | — | — | — | — | |
| Qwen3-0.6B-SWA-SinksAttention=sliding-window, Window Length=4k, Attention Sinks=128, Avg. Len.=370522026.02 | 23.67 | — | 13.08 | — | — | — | 8.56 | 25.71 | — | — | — | — | — | — | — | — | — | — | 23.52 | 16.01 | 69.41 | — | — | — | — | — | |
| MemoRAGCategory=Advanced RAG method2025.11 | 22.79 | — | 15.49 | — | 16.57 | 17.96 | 8.74 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 80 | 75 | 66.22 | 57.14 | 67.65 | |
| Qwen3-0.6B-MambaArchitecture=Hybrid Mamba-2, Avg. Len.=370522026.02 | 22.41 | — | 13.53 | — | — | — | 7.24 | 27.01 | — | — | — | — | — | — | — | — | — | — | 27.4 | 20.14 | 71.31 | — | — | — | — | — | |
| MsPoE2025.06 | 17.16 | 50.02 | 18.96 | 51.39 | 13.97 | 24.86 | 7.59 | 26.27 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PCD2025.06 | 15.29 | 51.09 | 20.31 | 50.11 | 16.47 | 27.13 | 6.7 | 26.87 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Base2025.06 | 15.22 | 46.72 | 20.03 | 51.27 | 15.5 | 26.26 | 6.87 | 25.98 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Rephrasing2025.06 | 13.28 | 45.13 | 18.94 | 49.53 | 13.22 | 28.7 | 6.25 | 25.02 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LUCID2026.02 | 8.6 | — | — | — | 27.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LUCID-PaTH2026.02 | 8.5 | — | — | — | 27.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SegR2025.06 | 8.31 | 4.86 | 4.18 | 27.41 | 10.13 | 26.41 | 3.94 | 12.18 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Diff2026.02 | 8.1 | — | — | — | 26.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PaTH2026.02 | 7.9 | — | — | — | 28.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Standard2026.02 | 7.3 | — | — | — | 24 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| C3Approach=Latent-based compression2025.12 | 7 | — | — | — | 9 | — | 8 | — | — | — | — | — | — | — | — | — | — | — | 2.08 | — | — | — | — | — | — | — | |
| GSA2026.02 | 5.1 | — | — | — | 23.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GLA2026.02 | 4.5 | — | — | — | 22.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GraphRAGCategory=Advanced RAG method2025.11 | 3.03 | — | 4.21 | — | 5.52 | 7.69 | 2.15 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 93.62 | 90.7 | 78.26 | 83.75 | 78.57 | |
| LightRAGCategory=Advanced RAG method2025.11 | 2.74 | — | 2.65 | — | 3.67 | 3.25 | 1.95 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 96.63 | 96.7 | 91.46 | 91.36 | 96.74 | |
| DeltaNet2026.02 | 1.9 | — | — | — | 3.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Best Published2026.03 | — | — | — | — | — | — | — | 63.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Claude Code + BM25Retriever=BM25, Base Model=Claude 3.5 Sonnet2026.03 | — | — | — | — | — | — | — | 62.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Codex (No Retriever)Retriever=None, Base Model=GPT-52026.03 | — | — | — | — | — | — | — | 61.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Codex + BM25Retriever=BM25, Base Model=GPT-52026.03 | — | — | — | — | — | — | — | 60.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Codex + Gemini Emb.Retriever=Gemini Embeddings, Base Model=GPT-52026.03 | — | — | — | — | — | — | — | 61.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GPT-5 Full ContextContext Type=Full Context2026.03 | — | — | — | — | — | — | — | 61 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| RAGRetriever=Gemini Embeddings2026.03 | — | — | — | — | — | — | — | 50.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ReAct AgentAgent Loop=ReAct2026.03 | — | — | — | — | — | — | — | 59 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| RLM2026.03 | — | — | — | — | — | — | — | 54 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |