Question Answering on NarrativeQA (EM, F1, Rate)
15EMNon-compression
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| Non-compressionBackbone LM=Llama-3.1-70B-Instruct, Category=Non-compression2025.10 | 15 | 35.2 | 1 | |
| ProLong-8BCategory=Long-context LLMs2025.10 | 14 | 31.63 | 1 | |
| Non-compressionBackbone LM=GPT-4.1-nano, Category=Non-compression2025.10 | 10.5 | 28.25 | 1 | |
| FILM-7BCategory=Long-context LLMs2025.10 | 10 | 28.47 | 1 | |
| BRIEF-PRO-AUTOBackbone LM=GPT-4.1-nano, Category=Abstractive2025.10 | 8.5 | 19.53 | 32 | |
| BRIEF-PRO-LOWBackbone LM=GPT-4.1-nano, Category=Abstractive2025.10 | 8.5 | 20.85 | 40 | |
| EXITBackbone LM=Llama-3.1-70B-Instruct, Category=Extractive2025.10 | 8 | 19.7 | 40 | |
| LongLLMLinguaBackbone LM=Llama-3.1-70B-Instruct, Category=Abstractive2025.10 | 8 | 21.09 | 17 | |
| BRIEF-PRO-AUTOBackbone LM=Llama-3.1-70B-Instruct, Category=Abstractive2025.10 | 8 | 22.83 | 32 | |
| Non-compressionBackbone LM=Llama-3.1-8B-Instruct, Category=Non-compression2025.10 | 7.5 | 29.43 | 1 | |
| Rerank Top-3Backbone LM=Llama-3.1-70B-Instruct, Category=Extractive2025.10 | 7.5 | 21.62 | 3 | |
| GPT-4.1-nanoBackbone LM=Llama-3.1-70B-Instruct, Category=Abstractive2025.10 | 7.5 | 22.91 | 91 | |
| BRIEF-PRO-LOWBackbone LM=Llama-3.1-70B-Instruct, Category=Abstractive2025.10 | 7.5 | 20.71 | 40 | |
| EXITBackbone LM=GPT-4.1-nano, Category=Extractive2025.10 | 7.5 | 18.45 | 40 | |
| Rerank Top-3Backbone LM=GPT-4.1-nano, Category=Extractive2025.10 | 7.5 | 19.1 | 3 | |
| GPT-4.1-nanoBackbone LM=GPT-4.1-nano, Category=Abstractive2025.10 | 7.5 | 24.5 | 91 | |
| BRIEF-PRO-HIGHBackbone LM=GPT-4.1-nano, Category=Abstractive2025.10 | 7.5 | 18.88 | 97 | |
| BRIEF-PRO-MEDIUMBackbone LM=GPT-4.1-nano, Category=Abstractive2025.10 | 7.5 | 19.25 | 66 | |
| BRIEF-PRO-LOWBackbone LM=Llama-3.1-8B-Instruct, Category=Abstractive2025.10 | 7 | 21.84 | 40 | |
| Llama-3.2-3B-InstructBackbone LM=Llama-3.1-70B-Instruct, Category=Abstractive2025.10 | 7 | 19.95 | 45 | |
| BRIEF-PRO-HIGHBackbone LM=Llama-3.1-70B-Instruct, Category=Abstractive2025.10 | 7 | 18.65 | 97 | |
| GPT-4.1-nanoBackbone LM=Llama-3.1-8B-Instruct, Category=Abstractive2025.10 | 6.5 | 24.46 | 91 | |
| BRIEF-PRO-HIGHBackbone LM=Llama-3.1-8B-Instruct, Category=Abstractive2025.10 | 6.5 | 19.1 | 97 | |
| BRIEF-PRO-MEDIUMBackbone LM=Llama-3.1-8B-Instruct, Category=Abstractive2025.10 | 6.5 | 20.4 | 66 | |
| BRIEF-PRO-MEDIUMBackbone LM=Llama-3.1-70B-Instruct, Category=Abstractive2025.10 | 6.5 | 20.02 | 66 | |
| RECOMP (Extractive)Backbone LM=Llama-3.1-70B-Instruct, Category=Extractive2025.10 | 6 | 16.62 | 94 | |
| EXITBackbone LM=Llama-3.1-8B-Instruct, Category=Extractive2025.10 | 5.5 | 20.11 | 40 | |
| Llama-3.2-3B-InstructBackbone LM=Llama-3.1-8B-Instruct, Category=Abstractive2025.10 | 5.5 | 19.81 | 45 | |
| BRIEF-PRO-AUTOBackbone LM=Llama-3.1-8B-Instruct, Category=Abstractive2025.10 | 5.5 | 21.73 | 32 | |
| RECOMP (Abstractive)Backbone LM=Llama-3.1-70B-Instruct, Category=Abstractive2025.10 | 5.5 | 17.16 | 17 | |
| RECOMP (Abstractive)Backbone LM=GPT-4.1-nano, Category=Abstractive2025.10 | 5.5 | 14.01 | 17 | |
| Llama-3.2-3B-InstructBackbone LM=GPT-4.1-nano, Category=Abstractive2025.10 | 5.5 | 17.15 | 45 | |
| Rerank Top-3Backbone LM=Llama-3.1-8B-Instruct, Category=Extractive2025.10 | 5 | 20.06 | 3 | |
| RECOMP (Extractive)Backbone LM=GPT-4.1-nano, Category=Extractive2025.10 | 5 | 15.62 | 94 | |
| RECOMP (Extractive)Backbone LM=Llama-3.1-8B-Instruct, Category=Extractive2025.10 | 4.5 | 16.19 | 94 | |
| LongLLMLinguaBackbone LM=Llama-3.1-8B-Instruct, Category=Abstractive2025.10 | 4.5 | 21.56 | 17 | |
| LongLLMLinguaBackbone LM=GPT-4.1-nano, Category=Abstractive2025.10 | 4 | 18.24 | 17 | |
| RECOMP (Abstractive)Backbone LM=Llama-3.1-8B-Instruct, Category=Abstractive2025.10 | 3.5 | 13.69 | 17 |