Question Answering on NQ (Natural Questions) (EM, F1)
42.5EMEXIT
Evaluation results
| Method | Links | ||
|---|---|---|---|
| EXITBackbone=Llama-3.1-70B-Instruct, Top-k Documents=Top-20, Compressor Type=Ext.2024.12 | 42.5 | 55.3 | |
| Original DocsBackbone=Llama-3.1-70B-Instruct, Top-k Documents=Top-20, Compressor Type=-2024.12 | 42.1 | 55 | |
| RECOMP-ExtrBackbone=Llama-3.1-70B-Instruct, Top-k Documents=Top-5, Compressor Type=Ext.2024.12 | 40.1 | 50.9 | |
| Original DocsBackbone=Llama-3.1-70B-Instruct, Top-k Documents=Top-5, Compressor Type=-2024.12 | 39.5 | 51.8 | |
| EXITBackbone=Llama-3.1-70B-Instruct, Top-k Documents=Top-5, Compressor Type=Ext.2024.12 | 39.5 | 51.9 | |
| Original DocsBackbone=Llama-3.1-8B-Instruct, Top-k Documents=Top-20, Compressor Type=-2024.12 | 39.2 | 51.4 | |
| EXITBackbone=Llama-3.1-8B-Instruct, Top-k Documents=Top-20, Compressor Type=Ext.2024.12 | 38.8 | 50.8 | |
| RECOMP-ExtrBackbone=Llama-3.1-70B-Instruct, Top-k Documents=Top-20, Compressor Type=Ext.2024.12 | 38.6 | 49.1 | |
| RefinerBackbone=Llama-3.1-70B-Instruct, Top-k Documents=Top-5, Compressor Type=Abs.2024.12 | 38.2 | 50.2 | |
| RECOMP-AbstBackbone=Llama-3.1-70B-Instruct, Top-k Documents=Top-5, Compressor Type=Abs.2024.12 | 38.1 | 50.3 | |
| CompActBackbone=Llama-3.1-70B-Instruct, Top-k Documents=Top-5, Compressor Type=Abs.2024.12 | 37.9 | 49.7 | |
| CompActBackbone=Llama-3.1-70B-Instruct, Top-k Documents=Top-20, Compressor Type=Abs.2024.12 | 37.6 | 49.1 | |
| RECOMP-AbstBackbone=Llama-3.1-70B-Instruct, Top-k Documents=Top-20, Compressor Type=Abs.2024.12 | 37.2 | 50 | |
| LongLLMLinguaBackbone=Llama-3.1-70B-Instruct, Top-k Documents=Top-20, Compressor Type=Ext.2024.12 | 37 | 49.1 | |
| Original DocsBackbone=Llama-3.1-8B-Instruct, Top-k Documents=Top-5, Compressor Type=-2024.12 | 36.9 | 48.8 | |
| RefinerBackbone=Llama-3.1-70B-Instruct, Top-k Documents=Top-20, Compressor Type=Abs.2024.12 | 36.5 | 47.6 | |
| RECOMP-ExtrBackbone=Llama-3.1-8B-Instruct, Top-k Documents=Top-5, Compressor Type=Ext.2024.12 | 35.9 | 45.8 | |
| EXITBackbone=Llama-3.1-8B-Instruct, Top-k Documents=Top-5, Compressor Type=Ext.2024.12 | 35.8 | 47.4 | |
| CompActBackbone=Llama-3.1-8B-Instruct, Top-k Documents=Top-20, Compressor Type=Abs.2024.12 | 35.6 | 47 | |
| LongLLMLinguaBackbone=Llama-3.1-70B-Instruct, Top-k Documents=Top-5, Compressor Type=Ext.2024.12 | 35.2 | 47.2 | |
| CompActBackbone=Llama-3.1-8B-Instruct, Top-k Documents=Top-5, Compressor Type=Abs.2024.12 | 35 | 46.3 | |
| RefinerBackbone=Llama-3.1-8B-Instruct, Top-k Documents=Top-5, Compressor Type=Abs.2024.12 | 34.4 | 46 | |
| RECOMP-ExtrBackbone=Llama-3.1-8B-Instruct, Top-k Documents=Top-20, Compressor Type=Ext.2024.12 | 34.2 | 43.7 | |
| RECOMP-AbstBackbone=Llama-3.1-8B-Instruct, Top-k Documents=Top-5, Compressor Type=Abs.2024.12 | 33.9 | 45.1 | |
| LongLLMLinguaBackbone=Llama-3.1-8B-Instruct, Top-k Documents=Top-20, Compressor Type=Ext.2024.12 | 33.8 | 45.2 | |
| RefinerBackbone=Llama-3.1-8B-Instruct, Top-k Documents=Top-20, Compressor Type=Abs.2024.12 | 32.2 | 43.1 | |
| LongLLMLinguaBackbone=Llama-3.1-8B-Instruct, Top-k Documents=Top-5, Compressor Type=Ext.2024.12 | 30.7 | 41.8 | |
| RECOMP-AbstBackbone=Llama-3.1-8B-Instruct, Top-k Documents=Top-20, Compressor Type=Abs.2024.12 | 30.2 | 40.7 |