Question Answering on NQ (EM, F1, Latency)
39.5EMOriginal Docs
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| Original DocsReader Model=Llama-3.1-70B-Instruct, Top-k Documents=Top-202024.12 | 39.5 | 52.5 | 25.8 | |
| EXITReader Model=Llama-3.1-70B-Instruct, Top-k Documents=Top-20, Compressor Type=Extractive2024.12 | 39.4 | 52.6 | 5.1 | |
| EXITReader Model=Llama-3.1-8B-Instruct, Top-k Documents=Top-20, Compressor Type=Extractive2024.12 | 38.1 | 50.8 | 1.8 | |
| EXITReader Model=Llama-3.1-70B-Instruct, Top-k Documents=Top-5, Compressor Type=Extractive2024.12 | 36.9 | 49.4 | 3.9 | |
| Original DocsReader Model=Llama-3.1-8B-Instruct, Top-k Documents=Top-202024.12 | 36.6 | 49.5 | 3.4 | |
| EXITReader Model=Llama-3.1-8B-Instruct, Top-k Documents=Top-5, Compressor Type=Extractive2024.12 | 35.9 | 47.8 | 0.8 | |
| RECOMP-ExtrReader Model=Llama-3.1-70B-Instruct, Top-k Documents=Top-5, Compressor Type=Extractive2024.12 | 35.8 | 45.3 | 2.5 | |
| Original DocsReader Model=Llama-3.1-70B-Instruct, Top-k Documents=Top-52024.12 | 35.6 | 48 | 8.6 | |
| RefinerReader Model=Llama-3.1-70B-Instruct, Top-k Documents=Top-5, Compressor Type=Abstractive2024.12 | 35.3 | 47.1 | 42.5 | |
| Original DocsReader Model=Llama-3.1-8B-Instruct, Top-k Documents=Top-52024.12 | 34.6 | 47.1 | 1 | |
| RECOMP-ExtrReader Model=Llama-3.1-8B-Instruct, Top-k Documents=Top-5, Compressor Type=Extractive2024.12 | 34.6 | 44.6 | 0.5 | |
| LongLLMLinguaReader Model=Llama-3.1-70B-Instruct, Top-k Documents=Top-20, Compressor Type=Extractive2024.12 | 34.5 | 46.4 | 10.9 | |
| RECOMP-AbstReader Model=Llama-3.1-70B-Instruct, Top-k Documents=Top-5, Compressor Type=Abstractive2024.12 | 34.1 | 47 | 4.5 | |
| CompActReader Model=Llama-3.1-70B-Instruct, Top-k Documents=Top-5, Compressor Type=Abstractive2024.12 | 34.1 | 45.4 | 11.9 | |
| CompActReader Model=Llama-3.1-70B-Instruct, Top-k Documents=Top-20, Compressor Type=Abstractive2024.12 | 33.9 | 45.1 | 30.8 | |
| CompActReader Model=Llama-3.1-8B-Instruct, Top-k Documents=Top-20, Compressor Type=Abstractive2024.12 | 33.8 | 45.4 | 26.1 | |
| RECOMP-ExtrReader Model=Llama-3.1-70B-Instruct, Top-k Documents=Top-20, Compressor Type=Extractive2024.12 | 33.6 | 42.7 | 2.4 | |
| LongLLMLinguaReader Model=Llama-3.1-8B-Instruct, Top-k Documents=Top-20, Compressor Type=Extractive2024.12 | 33.4 | 45.1 | 2.8 | |
| CompActReader Model=Llama-3.1-8B-Instruct, Top-k Documents=Top-5, Compressor Type=Abstractive2024.12 | 32.9 | 44.6 | 8.5 | |
| RefinerReader Model=Llama-3.1-8B-Instruct, Top-k Documents=Top-5, Compressor Type=Abstractive2024.12 | 32.9 | 45 | 28.1 | |
| RECOMP-ExtrReader Model=Llama-3.1-8B-Instruct, Top-k Documents=Top-20, Compressor Type=Extractive2024.12 | 32.8 | 42.6 | 0.6 | |
| RefinerReader Model=Llama-3.1-70B-Instruct, Top-k Documents=Top-20, Compressor Type=Abstractive2024.12 | 32.6 | 43.5 | 37.9 | |
| LongLLMLinguaReader Model=Llama-3.1-70B-Instruct, Top-k Documents=Top-5, Compressor Type=Extractive2024.12 | 32.2 | 44 | 4.4 | |
| RECOMP-AbstReader Model=Llama-3.1-70B-Instruct, Top-k Documents=Top-20, Compressor Type=Abstractive2024.12 | 31.5 | 45.1 | 4.5 | |
| RECOMP-AbstReader Model=Llama-3.1-8B-Instruct, Top-k Documents=Top-5, Compressor Type=Abstractive2024.12 | 31.3 | 43.2 | 1.6 | |
| LongLLMLinguaReader Model=Llama-3.1-8B-Instruct, Top-k Documents=Top-5, Compressor Type=Extractive2024.12 | 30.2 | 41.5 | 0.9 | |
| RefinerReader Model=Llama-3.1-8B-Instruct, Top-k Documents=Top-20, Compressor Type=Abstractive2024.12 | 30.1 | 41.4 | 28.7 | |
| RECOMP-AbstReader Model=Llama-3.1-8B-Instruct, Top-k Documents=Top-20, Compressor Type=Abstractive2024.12 | 26.9 | 38.3 | 1.7 |