Question Answering on NQ (Natural Questions) December 2018 Wikipedia dump (test)
36.9EMEXIT
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| EXITReader Model=Llama3.1-70B-Instruct, Compressor Type=Extractive2024.12 | 36.9 | 49.4 | 3.9 | |
| EXITReader Model=Llama3.1-8B-Instruct, Compressor Type=Extractive2024.12 | 35.9 | 47.8 | 0.8 | |
| RECOMP-ExtrReader Model=Llama3.1-70B-Instruct, Compressor Type=Extractive2024.12 | 35.8 | 45.3 | 2.5 | |
| Original DocsReader Model=Llama3.1-70B-Instruct, Compressor Type=None2024.12 | 35.6 | 48 | 8.6 | |
| RefinerReader Model=Llama3.1-70B-Instruct, Compressor Type=Abstractive2024.12 | 35.3 | 47.1 | 42.5 | |
| Original DocsReader Model=Llama3.1-8B-Instruct, Compressor Type=None2024.12 | 34.6 | 47.1 | 1 | |
| RECOMP-ExtrReader Model=Llama3.1-8B-Instruct, Compressor Type=Extractive2024.12 | 34.6 | 44.6 | 0.5 | |
| RECOMP-AbstReader Model=Llama3.1-70B-Instruct, Compressor Type=Abstractive2024.12 | 34.1 | 47 | 4.5 | |
| CompActReader Model=Llama3.1-70B-Instruct, Compressor Type=Abstractive2024.12 | 34.1 | 45.4 | 11.9 | |
| CompActReader Model=Llama3.1-8B-Instruct, Compressor Type=Abstractive2024.12 | 32.9 | 44.6 | 8.5 | |
| RefinerReader Model=Llama3.1-8B-Instruct, Compressor Type=Abstractive2024.12 | 32.9 | 45 | 28.1 | |
| LongLLMLinguaReader Model=Llama3.1-70B-Instruct, Compressor Type=Extractive2024.12 | 32.2 | 44 | 4.4 | |
| RECOMP-AbstReader Model=Llama3.1-8B-Instruct, Compressor Type=Abstractive2024.12 | 31.3 | 43.2 | 1.6 | |
| LongLLMLinguaReader Model=Llama3.1-8B-Instruct, Compressor Type=Extractive2024.12 | 30.2 | 41.5 | 0.9 |