Question Answering on TriviaQA (TQA)
71.1EMOriginal Docs
Evaluation Results
| Method | Links | ||||
|---|---|---|---|---|---|
| Original DocsBackbone=Llama-3.1-70B-Instruct, Top-k Documents=Top-20, Compressor Type=-2024.12 | 71.1 | 79.1 | — | — | |
| EXITBackbone=Llama-3.1-70B-Instruct, Top-k Documents=Top-20, Compressor Type=Ext.2024.12 | 71 | 79 | — | — | |
| LongLLMLinguaBackbone=Llama-3.1-70B-Instruct, Top-k Documents=Top-20, Compressor Type=Ext.2024.12 | 70.5 | 78.5 | — | — | |
| EXITBackbone=Llama-3.1-70B-Instruct, Top-k Documents=Top-5, Compressor Type=Ext.2024.12 | 69.6 | 77.8 | — | — | |
| LongLLMLinguaBackbone=Llama-3.1-70B-Instruct, Top-k Documents=Top-5, Compressor Type=Ext.2024.12 | 69.3 | 77.2 | — | — | |
| Original DocsReader Model=Llama-3.1-70B-Instruct, Top-k Documents=Top-202024.12 | 69.1 | 77.6 | 24.9 | — | |
| EXITReader Model=Llama-3.1-70B-Instruct, Top-k Documents=Top-20, Compressor Type=Extractive2024.12 | 68.7 | 77.3 | 4.3 | — | |
| RECOMP-ExtrBackbone=Llama-3.1-70B-Instruct, Top-k Documents=Top-20, Compressor Type=Ext.2024.12 | 68.4 | 75.6 | — | — | |
| Original DocsBackbone=Llama-3.1-70B-Instruct, Top-k Documents=Top-5, Compressor Type=-2024.12 | 68.3 | 76.5 | — | — | |
| LongLLMLinguaReader Model=Llama-3.1-70B-Instruct, Top-k Documents=Top-20, Compressor Type=Extractive2024.12 | 68.2 | 76.9 | 10.4 | — | |
| RECOMP-ExtrBackbone=Llama-3.1-70B-Instruct, Top-k Documents=Top-5, Compressor Type=Ext.2024.12 | 68.1 | 75.6 | — | — | |
| CompActBackbone=Llama-3.1-70B-Instruct, Top-k Documents=Top-5, Compressor Type=Abs.2024.12 | 67.7 | 76 | — | — | |
| RefinerBackbone=Llama-3.1-70B-Instruct, Top-k Documents=Top-5, Compressor Type=Abs.2024.12 | 67.7 | 76.1 | — | — | |
| EXITReader Model=Llama-3.1-70B-Instruct, Top-k Documents=Top-5, Compressor Type=Extractive2024.12 | 67.3 | 75.9 | 3.1 | — | |
| LongLLMLinguaReader Model=Llama-3.1-70B-Instruct, Top-k Documents=Top-5, Compressor Type=Extractive2024.12 | 66.7 | 75.2 | 3.9 | — | |
| RefinerBackbone=Llama-3.1-70B-Instruct, Top-k Documents=Top-20, Compressor Type=Abs.2024.12 | 66.6 | 74.7 | — | — | |
| CompActBackbone=Llama-3.1-70B-Instruct, Top-k Documents=Top-20, Compressor Type=Abs.2024.12 | 66.4 | 74.4 | — | — | |
| RECOMP-AbstBackbone=Llama-3.1-70B-Instruct, Top-k Documents=Top-20, Compressor Type=Abs.2024.12 | 65.6 | 74 | — | — | |
| Original DocsReader Model=Llama-3.1-70B-Instruct, Top-k Documents=Top-52024.12 | 65.1 | 73.9 | 7.7 | — | |
| RefinerReader Model=Llama-3.1-70B-Instruct, Top-k Documents=Top-5, Compressor Type=Abstractive2024.12 | 64.3 | 73 | 18.3 | — | |
| Original DocsBackbone=Llama-3.1-8B-Instruct, Top-k Documents=Top-20, Compressor Type=-2024.12 | 64.2 | 73.2 | — | — | |
| LongLLMLinguaBackbone=Llama-3.1-8B-Instruct, Top-k Documents=Top-20, Compressor Type=Ext.2024.12 | 63.8 | 72.1 | — | — | |
| RECOMP-ExtrReader Model=Llama-3.1-70B-Instruct, Top-k Documents=Top-5, Compressor Type=Extractive2024.12 | 63.5 | 71 | 2.2 | — | |
| EXITBackbone=Llama-3.1-8B-Instruct, Top-k Documents=Top-20, Compressor Type=Ext.2024.12 | 63.4 | 72.3 | — | — | |
| RECOMP-AbstBackbone=Llama-3.1-70B-Instruct, Top-k Documents=Top-5, Compressor Type=Abs.2024.12 | 63.4 | 72.4 | — | — | |
| RECOMP-AbstReader Model=Llama-3.1-70B-Instruct, Top-k Documents=Top-20, Compressor Type=Abstractive2024.12 | 63.4 | 72.2 | 3.7 | — | |
| RECOMP-ExtrReader Model=Llama-3.1-70B-Instruct, Top-k Documents=Top-20, Compressor Type=Extractive2024.12 | 63.1 | 70.3 | 2.2 | — | |
| RefinerReader Model=Llama-3.1-70B-Instruct, Top-k Documents=Top-20, Compressor Type=Abstractive2024.12 | 62.9 | 71.4 | 48.9 | — | |
| EXITReader Model=Llama-3.1-8B-Instruct, Top-k Documents=Top-20, Compressor Type=Extractive2024.12 | 62.8 | 72 | 1.7 | — | |
| CompActReader Model=Llama-3.1-70B-Instruct, Top-k Documents=Top-5, Compressor Type=Abstractive2024.12 | 62.6 | 71.1 | 11.7 | — | |
| LongLLMLinguaReader Model=Llama-3.1-8B-Instruct, Top-k Documents=Top-20, Compressor Type=Extractive2024.12 | 62.4 | 71.2 | 2.7 | — | |
| Original DocsReader Model=Llama-3.1-8B-Instruct, Top-k Documents=Top-202024.12 | 62 | 71.7 | 2.9 | — | |
| CompActReader Model=Llama-3.1-70B-Instruct, Top-k Documents=Top-20, Compressor Type=Abstractive2024.12 | 61.7 | 70 | 28.1 | — | |
| Original DocsBackbone=Llama-3.1-8B-Instruct, Top-k Documents=Top-5, Compressor Type=-2024.12 | 61.5 | 70.3 | — | — | |
| RECOMP-AbstReader Model=Llama-3.1-70B-Instruct, Top-k Documents=Top-5, Compressor Type=Abstractive2024.12 | 61.3 | 70.6 | 3.3 | — | |
| RefinerBackbone=Llama-3.1-8B-Instruct, Top-k Documents=Top-5, Compressor Type=Abs.2024.12 | 61 | 70 | — | — | |
| EXITBackbone=Llama-3.1-8B-Instruct, Top-k Documents=Top-5, Compressor Type=Ext.2024.12 | 61 | 69.8 | — | — | |
| LongLLMLinguaBackbone=Llama-3.1-8B-Instruct, Top-k Documents=Top-5, Compressor Type=Ext.2024.12 | 60.8 | 68.8 | — | — | |
| EXITReader Model=Llama-3.1-8B-Instruct, Top-k Documents=Top-5, Compressor Type=Extractive2024.12 | 60.8 | 69.9 | 0.7 | — | |
| CompActBackbone=Llama-3.1-8B-Instruct, Top-k Documents=Top-5, Compressor Type=Abs.2024.12 | 60.3 | 69.2 | — | — | |
| CompActBackbone=Llama-3.1-8B-Instruct, Top-k Documents=Top-20, Compressor Type=Abs.2024.12 | 60.1 | 69.2 | — | — | |
| RefinerBackbone=Llama-3.1-8B-Instruct, Top-k Documents=Top-20, Compressor Type=Abs.2024.12 | 59.6 | 68.6 | — | — | |
| LongLLMLinguaReader Model=Llama-3.1-8B-Instruct, Top-k Documents=Top-5, Compressor Type=Extractive2024.12 | 59.4 | 68 | 0.8 | — | |
| RECOMP-AbstBackbone=Llama-3.1-8B-Instruct, Top-k Documents=Top-20, Compressor Type=Abs.2024.12 | 59.3 | 67.6 | — | — | |
| RefinerReader Model=Llama-3.1-8B-Instruct, Top-k Documents=Top-5, Compressor Type=Abstractive2024.12 | 59.2 | 68.9 | 10.9 | — | |
| Original DocsReader Model=Llama-3.1-8B-Instruct, Top-k Documents=Top-52024.12 | 58.8 | 68.6 | 0.9 | — | |
| RECOMP-ExtrBackbone=Llama-3.1-8B-Instruct, Top-k Documents=Top-5, Compressor Type=Ext.2024.12 | 58.5 | 66.1 | — | — | |
| CompActReader Model=Llama-3.1-8B-Instruct, Top-k Documents=Top-5, Compressor Type=Abstractive2024.12 | 58.1 | 67.7 | 8.8 | — | |
| RECOMP-AbstBackbone=Llama-3.1-8B-Instruct, Top-k Documents=Top-5, Compressor Type=Abs.2024.12 | 57.8 | 66.7 | — | — | |
| CompActReader Model=Llama-3.1-8B-Instruct, Top-k Documents=Top-20, Compressor Type=Abstractive2024.12 | 57.8 | 67.5 | 24.5 | — | |
| RefinerReader Model=Llama-3.1-8B-Instruct, Top-k Documents=Top-20, Compressor Type=Abstractive2024.12 | 57.6 | 67 | 44.2 | — | |
| RECOMP-ExtrBackbone=Llama-3.1-8B-Instruct, Top-k Documents=Top-20, Compressor Type=Ext.2024.12 | 57.5 | 64.9 | — | — | |
| RECOMP-AbstReader Model=Llama-3.1-8B-Instruct, Top-k Documents=Top-20, Compressor Type=Abstractive2024.12 | 57.3 | 66.6 | 1.9 | — | |
| RECOMP-ExtrReader Model=Llama-3.1-8B-Instruct, Top-k Documents=Top-5, Compressor Type=Extractive2024.12 | 56.5 | 65.1 | 0.4 | — | |
| RECOMP-AbstReader Model=Llama-3.1-8B-Instruct, Top-k Documents=Top-5, Compressor Type=Abstractive2024.12 | 55.9 | 65.7 | 1.4 | — | |
| RECOMP-ExtrReader Model=Llama-3.1-8B-Instruct, Top-k Documents=Top-20, Compressor Type=Extractive2024.12 | 55.5 | 63.6 | 0.4 | — | |
| Gemma 3 27B IT2026.03 | — | — | — | 64.87 | |
| HiPRAGBackbone=Qwen2.5-7B2025.10 | — | — | — | 62.6 | |
| OPTIMERBase Model=SEA-LION v42026.03 | — | — | — | 45.41 | |
| OPTIMERBase Model=Gemma 3 27B2026.03 | — | — | — | 53 | |
| ReSeekBackbone=Qwen2.5-7B2025.10 | — | — | — | 64 | |
| SEA-LION v4 27B IT2026.03 | — | — | — | 57.77 |