Question Answering on QASPER (test)
61.5F1 Score (Match)RAGA Fusion (C1200)
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| RAGA Fusion (C1200)Training=Zero-shot, Context size=12002026.05 | 61.5 | — | — | — | 41.1 | |
| Human lower boundTraining=-2026.05 | 60.9 | — | — | — | 71.6 | |
| RAGA KG (C1500)Training=Zero-shot, Context size=15002026.05 | 60.5 | — | — | — | 38.8 | |
| DTCRS + GPT-4Language Model=GPT-42026.04 | 58.5 | — | — | — | — | |
| RAPTORLanguage Model=GPT-42024.01 | 55.7 | — | — | — | — | |
| RAPTOR + GPT-4Language Model=GPT-42026.04 | 55.7 | — | — | — | — | |
| No-KG Control (FAISS)Training=Zero-shot2026.05 | 55.4 | — | — | — | 35.9 | |
| CoLT5 XLModel size=XL2024.01 | 53.9 | — | — | — | — | |
| CoLT5 XL2026.04 | 53.9 | — | — | — | — | |
| LongT5 XLModel size=XL2024.01 | 53.1 | — | — | — | — | |
| LongT5 XL2026.04 | 53.1 | — | — | — | — | |
| DPR + GPT-4Language Model=GPT-42026.04 | 51.3 | — | — | — | — | |
| Gold EvidenceContext Setting=Gold Evidence, Avg Context Length=129 words, Generation Model=Llama3.1-8B-Instruct2025.05 | 50.71 | — | — | — | — | |
| DISRetrievalContext Setting=Full Document + Retrieved Evidence, Token Budget=400, Embedding Model=OpenAI, Generation Model=Llama3.1-8B-Instruct2025.05 | 50.2 | — | — | — | — | |
| DISRetrievalContext Setting=Full Document + Retrieved Evidence, Token Budget=300, Embedding Model=OpenAI, Generation Model=Llama3.1-8B-Instruct2025.05 | 50.11 | — | — | — | — | |
| DISRetrievalContext Setting=Full Document + Retrieved Evidence, Token Budget=200, Embedding Model=OpenAI, Generation Model=Llama3.1-8B-Instruct2025.05 | 50.05 | — | — | — | — | |
| DISRetrievalContext Setting=Full Document + Retrieved Evidence, Token Budget=300, Embedding Model=SBERT, Generation Model=Llama3.1-8B-Instruct2025.05 | 49.77 | — | — | — | — | |
| PECAN2024.10 | 49.7 | 47.9 | 66.9 | 1 | — | |
| DISRetrievalContext Setting=Full Document + Retrieved Evidence, Token Budget=200, Embedding Model=SBERT, Generation Model=Llama3.1-8B-Instruct2025.05 | 49.65 | — | — | — | — | |
| DISRetrievalContext Setting=Full Document + Retrieved Evidence, Token Budget=400, Embedding Model=SBERT, Generation Model=Llama3.1-8B-Instruct2025.05 | 49.54 | — | — | — | — | |
| Llama-3.1-8B2024.10 | 49.4 | 47.6 | 92.5 | 1.38 | — | |
| Full DocumentContext Setting=Full Document, Avg Context Length=4170 words, Generation Model=Llama3.1-8B-Instruct2025.05 | 48.81 | — | — | — | — | |
| BM25Top-k=142024.10 | 47 | 45.1 | 69.3 | 1.04 | — | |
| DragonTop-k=142024.10 | 46.9 | 44.8 | 67 | 1 | — | |
| RAPTOR-CTStrategy=collapsed tree, Top-k=422024.10 | 46.9 | 44.7 | 67.3 | 1.01 | — | |
| SBERTTop-k=142024.10 | 46.6 | 44.5 | 68.9 | 1.03 | — | |
| DISRetrievalBase Model=GPT-4.1-mini, Context Length=400, Embedding Model=OpenAI text-embedding-3-large2025.05 | 46.31 | — | — | — | — | |
| flatten-sentenceBase Model=GPT-4.1-mini, Context Length=400, Embedding Model=OpenAI text-embedding-3-large2025.05 | 45.78 | — | — | — | — | |
| BisectionBase Model=GPT-4.1-mini, Context Length=400, Embedding Model=OpenAI text-embedding-3-large2025.05 | 45.69 | — | — | — | — | |
| flatten-sentenceBase Model=GPT-4.1-mini, Context Length=300, Embedding Model=OpenAI text-embedding-3-large2025.05 | 45.28 | — | — | — | — | |
| DISRetrievalBase Model=GPT-4.1-mini, Context Length=300, Embedding Model=OpenAI text-embedding-3-large2025.05 | 45.19 | — | — | — | — | |
| DISRetrievalBase Model=GPT-4.1-mini, Context Length=400, Embedding Model=SBERT2025.05 | 44.95 | — | — | — | — | |
| flatten-chunkBase Model=GPT-4.1-mini, Context Length=400, Embedding Model=OpenAI text-embedding-3-large2025.05 | 44.78 | — | — | — | — | |
| RAPTOR-CTStrategy=collapsed tree, Top-k=52024.10 | 44.6 | 42.7 | 16.6 | 0.25 | — | |
| BisectionBase Model=GPT-4.1-mini, Context Length=300, Embedding Model=OpenAI text-embedding-3-large2025.05 | 44.54 | — | — | — | — | |
| SBERTTop-k=52024.10 | 44.4 | 42.4 | 26 | 0.39 | — | |
| RAPTORBase Model=GPT-4.1-mini, Context Length=400, Embedding Model=OpenAI text-embedding-3-large2025.05 | 43.85 | — | — | — | — | |
| DISRetrievalBase Model=Deepseek-v3, Context Length=400, Embedding Model=OpenAI text-embedding-3-large2025.05 | 43.65 | — | — | — | — | |
| BisectionBase Model=Deepseek-v3, Context Length=400, Embedding Model=OpenAI text-embedding-3-large2025.05 | 43.57 | — | — | — | — | |
| BisectionBase Model=GPT-4.1-mini, Context Length=400, Embedding Model=SBERT2025.05 | 43.49 | — | — | — | — | |
| DISRetrievalBase Model=GPT-4.1-mini, Context Length=200, Embedding Model=OpenAI text-embedding-3-large2025.05 | 43.45 | — | — | — | — | |
| flatten-chunkBase Model=GPT-4.1-mini, Context Length=300, Embedding Model=OpenAI text-embedding-3-large2025.05 | 43.34 | — | — | — | — | |
| RAPTORBase Model=GPT-4.1-mini, Context Length=300, Embedding Model=OpenAI text-embedding-3-large2025.05 | 43.26 | — | — | — | — | |
| LongLLMLingua2024.10 | 43.2 | 43 | 159.7 | 2.39 | — | |
| BisectionBase Model=GPT-4.1-mini, Context Length=200, Embedding Model=OpenAI text-embedding-3-large2025.05 | 43.12 | — | — | — | — | |
| flatten-sentenceBase Model=GPT-4.1-mini, Context Length=200, Embedding Model=OpenAI text-embedding-3-large2025.05 | 43.08 | — | — | — | — | |
| flatten-sentenceBase Model=Deepseek-v3, Context Length=400, Embedding Model=OpenAI text-embedding-3-large2025.05 | 43.04 | — | — | — | — | |
| DISRetrievalBase Model=GPT-4.1-mini, Context Length=300, Embedding Model=SBERT2025.05 | 43.01 | — | — | — | — | |
| DragonTop-k=52024.10 | 43 | 41.4 | 24.5 | 0.36 | — | |
| BisectionBase Model=GPT-4.1-mini, Context Length=300, Embedding Model=SBERT2025.05 | 42.74 | — | — | — | — | |
| flatten-chunkBase Model=GPT-4.1-mini, Context Length=400, Embedding Model=SBERT2025.05 | 42.72 | — | — | — | — | |
| RAPTORBase Model=GPT-4.1-mini, Context Length=400, Embedding Model=SBERT2025.05 | 42.5 | — | — | — | — | |
| flatten-sentenceBase Model=GPT-4.1-mini, Context Length=400, Embedding Model=SBERT2025.05 | 42.44 | — | — | — | — | |
| flatten-chunkBase Model=Deepseek-v3, Context Length=400, Embedding Model=OpenAI text-embedding-3-large2025.05 | 42.26 | — | — | — | — | |
| DISRetrievalBase Model=Deepseek-v3, Context Length=300, Embedding Model=OpenAI text-embedding-3-large2025.05 | 42.19 | — | — | — | — | |
| RAPTORBase Model=Deepseek-v3, Context Length=400, Embedding Model=OpenAI text-embedding-3-large2025.05 | 42.17 | — | — | — | — | |
| RAPTOR-TTStrategy=tree traversal2024.10 | 42.1 | 40.1 | 17.7 | 0.26 | — | |
| flatten-sentenceBase Model=GPT-4.1-mini, Context Length=300, Embedding Model=SBERT2025.05 | 41.84 | — | — | — | — | |
| BisectionBase Model=Deepseek-v3, Context Length=300, Embedding Model=OpenAI text-embedding-3-large2025.05 | 41.81 | — | — | — | — | |
| flatten-sentenceBase Model=Deepseek-v3, Context Length=300, Embedding Model=OpenAI text-embedding-3-large2025.05 | 41.57 | — | — | — | — | |
| DISRetrievalBase Model=Deepseek-v3, Context Length=400, Embedding Model=SBERT2025.05 | 41.51 | — | — | — | — | |
| flatten-chunkBase Model=GPT-4.1-mini, Context Length=300, Embedding Model=SBERT2025.05 | 41.38 | — | — | — | — | |
| flatten-chunkBase Model=GPT-4.1-mini, Context Length=200, Embedding Model=OpenAI text-embedding-3-large2025.05 | 41.13 | — | — | — | — | |
| BM25Top-k=52024.10 | 41 | 39.6 | 26.3 | 0.39 | — | |
| DISRetrievalBase Model=GPT-4.1-mini, Context Length=200, Embedding Model=SBERT2025.05 | 40.99 | — | — | — | — | |
| BisectionBase Model=GPT-4.1-mini, Context Length=200, Embedding Model=SBERT2025.05 | 40.98 | — | — | — | — | |
| BisectionBase Model=Deepseek-v3, Context Length=400, Embedding Model=SBERT2025.05 | 40.96 | — | — | — | — | |
| RAPTORBase Model=GPT-4.1-mini, Context Length=200, Embedding Model=OpenAI text-embedding-3-large2025.05 | 40.88 | — | — | — | — | |
| flatten-sentenceBase Model=Deepseek-v3, Context Length=400, Embedding Model=SBERT2025.05 | 40.82 | — | — | — | — | |
| flatten-chunkBase Model=Deepseek-v3, Context Length=300, Embedding Model=OpenAI text-embedding-3-large2025.05 | 40.78 | — | — | — | — | |
| DISRetrievalBase Model=UnifiedQA-3B, Context Length=400, Embedding Model=OpenAI text-embedding-3-large2025.05 | 40.74 | — | — | — | — | |
| DISRetrievalBase Model=UnifiedQA-3B, Context Length=400, Embedding Model=SBERT2025.05 | 40.65 | — | — | — | — | |
| flatten-chunkBase Model=Deepseek-v3, Context Length=400, Embedding Model=SBERT2025.05 | 40.57 | — | — | — | — | |
| RAPTORBase Model=Deepseek-v3, Context Length=300, Embedding Model=OpenAI text-embedding-3-large2025.05 | 40.5 | — | — | — | — | |
| DISRetrievalBase Model=Deepseek-v3, Context Length=300, Embedding Model=SBERT2025.05 | 40.39 | — | — | — | — | |
| flatten-sentenceBase Model=UnifiedQA-3B, Context Length=400, Embedding Model=OpenAI text-embedding-3-large2025.05 | 40.06 | — | — | — | — | |
| DISRetrievalBase Model=UnifiedQA-3B, Context Length=300, Embedding Model=OpenAI text-embedding-3-large2025.05 | 40.03 | — | — | — | — | |
| RAPTORBase Model=UnifiedQA-3B, Context Length=300, Embedding Model=OpenAI text-embedding-3-large2025.05 | 39.96 | — | — | — | — | |
| RAPTORBase Model=GPT-4.1-mini, Context Length=300, Embedding Model=SBERT2025.05 | 39.95 | — | — | — | — | |
| flatten-sentenceBase Model=GPT-4.1-mini, Context Length=200, Embedding Model=SBERT2025.05 | 39.82 | — | — | — | — | |
| BisectionBase Model=Deepseek-v3, Context Length=300, Embedding Model=SBERT2025.05 | 39.8 | — | — | — | — | |
| RAPTORBase Model=Deepseek-v3, Context Length=400, Embedding Model=SBERT2025.05 | 39.77 | — | — | — | — | |
| DISRetrievalBase Model=Deepseek-v3, Context Length=200, Embedding Model=OpenAI text-embedding-3-large2025.05 | 39.77 | — | — | — | — | |
| BisectionBase Model=UnifiedQA-3B, Context Length=400, Embedding Model=OpenAI text-embedding-3-large2025.05 | 39.7 | — | — | — | — | |
| DISRetrievalBase Model=UnifiedQA-3B, Context Length=300, Embedding Model=SBERT2025.05 | 39.56 | — | — | — | — | |
| RAPTORBase Model=UnifiedQA-3B, Context Length=400, Embedding Model=OpenAI text-embedding-3-large2025.05 | 39.53 | — | — | — | — | |
| BisectionBase Model=UnifiedQA-3B, Context Length=300, Embedding Model=OpenAI text-embedding-3-large2025.05 | 39.49 | — | — | — | — | |
| flatten-sentenceBase Model=Deepseek-v3, Context Length=300, Embedding Model=SBERT2025.05 | 39.41 | — | — | — | — | |
| flatten-sentenceBase Model=UnifiedQA-3B, Context Length=300, Embedding Model=OpenAI text-embedding-3-large2025.05 | 39.31 | — | — | — | — | |
| flatten-sentenceBase Model=Deepseek-v3, Context Length=200, Embedding Model=OpenAI text-embedding-3-large2025.05 | 39.27 | — | — | — | — | |
| BisectionBase Model=Deepseek-v3, Context Length=200, Embedding Model=OpenAI text-embedding-3-large2025.05 | 39.27 | — | — | — | — | |
| flatten-chunkBase Model=UnifiedQA-3B, Context Length=400, Embedding Model=OpenAI text-embedding-3-large2025.05 | 39.03 | — | — | — | — | |
| MeMWalker2024.10 | 39 | 36.8 | 123.9 | 1.85 | — | |
| BisectionBase Model=UnifiedQA-3B, Context Length=400, Embedding Model=SBERT2025.05 | 38.84 | — | — | — | — | |
| DISRetrievalBase Model=UnifiedQA-3B, Context Length=200, Embedding Model=OpenAI text-embedding-3-large2025.05 | 38.49 | — | — | — | — | |
| flatten-chunkBase Model=UnifiedQA-3B, Context Length=300, Embedding Model=OpenAI text-embedding-3-large2025.05 | 38.46 | — | — | — | — | |
| flatten-sentenceBase Model=UnifiedQA-3B, Context Length=200, Embedding Model=OpenAI text-embedding-3-large2025.05 | 38.43 | — | — | — | — | |
| flatten-chunkBase Model=Deepseek-v3, Context Length=200, Embedding Model=OpenAI text-embedding-3-large2025.05 | 38.14 | — | — | — | — | |
| flatten-sentenceBase Model=UnifiedQA-3B, Context Length=400, Embedding Model=SBERT2025.05 | 37.99 | — | — | — | — | |
| flatten-chunkBase Model=Deepseek-v3, Context Length=300, Embedding Model=SBERT2025.05 | 37.99 | — | — | — | — |