Question Answering on NarrativeQA (F1, Avg Token Count)
28.94F1 ScoreBaseline
Evaluation results
| Method | Links | ||
|---|---|---|---|
| BaselineLLM Backbone=GPT-4o mini, Context Type=Full Context2026.02 | 28.94 | 29,520.4 | |
| BaselineLLM Backbone=Llama-3.1 8B, Context Type=Full Context2026.02 | 24.97 | 29,883.02 | |
| RAG with AttentionRetriever-LlamaLLM Backbone=GPT-4o mini, Retrieval Method=AttentionRetriever-Llama2026.02 | 23.93 | 284.1 | |
| RAG with AttentionRetriever-QwenLLM Backbone=GPT-4o mini, Retrieval Method=AttentionRetriever-Qwen2026.02 | 20.76 | 312.4 | |
| RAG with SPScannerLLM Backbone=GPT-4o mini, Retrieval Method=SPScanner2026.02 | 20.62 | 325.47 | |
| RAG with AttentionRetriever-LlamaLLM Backbone=Qwen-2.5 7B, Retrieval Method=AttentionRetriever-Llama2026.02 | 17.46 | 299.74 | |
| RAG with AttentionRetriever-LlamaLLM Backbone=Llama-3.1 8B, Retrieval Method=AttentionRetriever-Llama2026.02 | 16.54 | 322.09 | |
| RAG with AttentionRetriever-QwenLLM Backbone=Qwen-2.5 7B, Retrieval Method=AttentionRetriever-Qwen2026.02 | 15.32 | 328.52 | |
| RAG with SPScannerLLM Backbone=Qwen-2.5 7B, Retrieval Method=SPScanner2026.02 | 15.25 | 341.11 | |
| RAG with SPScannerLLM Backbone=Llama-3.1 8B, Retrieval Method=SPScanner2026.02 | 15.06 | 363.54 | |
| BaselineLLM Backbone=Qwen-2.5 7B, Context Type=Full Context2026.02 | 14.22 | 29,920.98 | |
| RAG with AttentionRetriever-QwenLLM Backbone=Llama-3.1 8B, Retrieval Method=AttentionRetriever-Qwen2026.02 | 14.08 | 350.92 | |
| RAG with AttentionRetriever-LlamaLLM Backbone=Mistral-7B v0.3, Retrieval Method=AttentionRetriever-Llama2026.02 | 13.21 | 316.245 | |
| RAG with AttentionRetriever-QwenLLM Backbone=Mistral-7B v0.3, Retrieval Method=AttentionRetriever-Qwen2026.02 | 10.73 | 347.56 | |
| RAG with SPScannerLLM Backbone=Mistral-7B v0.3, Retrieval Method=SPScanner2026.02 | 9.97 | 359.56 | |
| BaselineLLM Backbone=Mistral-7B v0.3, Context Type=Full Context2026.02 | 7.24 | 35,277.56 |