Open-Domain Question Answering on WQ (test)
33.71EMECoRAG
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| ECoRAGReader Model=Flan-UL2, #tokens=6712025.06 | 33.71 | 46.08 | — | |
| CompActReader Model=Flan-UL2, #tokens=2182025.06 | 33.07 | 44.45 | — | |
| closed-bookReader Model=Flan-UL2, #tokens=02025.06 | 32.97 | 42.33 | — | |
| LongLLMLinguaReader Model=Flan-UL2, #tokens=16292025.06 | 32.78 | 45.27 | — | |
| RECOMP (extractive)Reader Model=Flan-UL2, #tokens=8572025.06 | 31.25 | 43.18 | — | |
| RECOMP (abstractive)Reader Model=Flan-UL2, #tokens=1572025.06 | 31.15 | 38.56 | — | |
| LLMLinguaReader Model=Flan-UL2, #tokens=8682025.06 | 31.1 | 40.87 | — | |
| LLMLingua-2Reader Model=Flan-UL2, #tokens=15802025.06 | 30.61 | 41.76 | — | |
| ECoRAG#tokens=560, Backbone=GPT-4o-mini2025.06 | 30.17 | 46.13 | — | |
| ECoRAG#tokens=5602025.06 | 30.17 | 46.13 | — | |
| CompAct#tokens=752025.06 | 29.77 | 44.25 | — | |
| ECoRAGReader model=Gemma2, Compression strategy=RAG with 100 documents compressed2025.06 | 27.41 | 44 | 504 | |
| LongLLMLinguaReader model=Gemma2, Compression strategy=RAG with 100 documents compressed2025.06 | 27.02 | 44.23 | 1,399 | |
| closed-bookReader model=Gemma2, Compression=None2025.06 | 26.77 | 43.24 | 0 | |
| CompActReader model=Gemma2, Compression strategy=RAG with 100 documents compressed2025.06 | 26.67 | 43.04 | 78 | |
| RECOMP (extractive)Reader model=Gemma2, Compression strategy=RAG with 100 documents compressed2025.06 | 26.03 | 42.22 | 680 | |
| RECOMP (abstractive)Reader model=Gemma2, Compression strategy=RAG with 100 documents compressed2025.06 | 25.2 | 41.6 | 21 | |
| LLMLinguaReader model=Gemma2, Compression strategy=RAG with 100 documents compressed2025.06 | 25.04 | 42.08 | 649 | |
| closed-book#tokens=02025.06 | 24.51 | 42.73 | — | |
| LLMLingua-2Reader model=Gemma2, Compression strategy=RAG with 100 documents compressed2025.06 | 24.36 | 40.52 | 1,401 | |
| LLMLingua-2#tokens=11132025.06 | 23.52 | 40.61 | — | |
| LLMLingua#tokens=6412025.06 | 22.98 | 39.77 | — | |
| closed-bookReader Model=Llama3, Retrieved Documents=0, Compression=None2025.06 | 21.79 | 35.81 | 0 | |
| ECoRAGReader Model=Llama3, Retrieved Documents=100, Compression=Compressed2025.06 | 21.6 | 35.43 | 588 | |
| LongLLMLingua#tokens=13552025.06 | 21.51 | 39.13 | — | |
| CompActReader Model=Llama3, Retrieved Documents=100, Compression=Compressed2025.06 | 21.41 | 35.21 | 76 | |
| standard RAG (100 documents)#tokens=137312025.06 | 21.11 | 38.72 | — | |
| BGE-M3Top-k retrieval=10, #tokens=322, Backbone=GPT-4o-mini2025.06 | 20.77 | 38.27 | — | |
| BGE-M3Top-k retrieval=20, #tokens=645, Backbone=GPT-4o-mini2025.06 | 20.77 | 38 | — | |
| RECOMP (abstractive)#tokens=192025.06 | 20.47 | 36.9 | — | |
| RECOMP (extractive)#tokens=6582025.06 | 19.54 | 36.83 | — | |
| RECOMP (abstractive)Reader Model=Llama3, Retrieved Documents=100, Compression=Compressed2025.06 | 18.8 | 33.25 | 21 | |
| LongLLMLinguaReader Model=Llama3, Retrieved Documents=100, Compression=Compressed2025.06 | 18.7 | 32.12 | 1,372 | |
| LLMLinguaReader Model=Llama3, Retrieved Documents=100, Compression=Compressed2025.06 | 17.62 | 30.92 | 646 | |
| RECOMP (extractive)Reader Model=Llama3, Retrieved Documents=100, Compression=Compressed2025.06 | 16.19 | 27.8 | 667 | |
| LLMLingua-2Reader Model=Llama3, Retrieved Documents=100, Compression=Compressed2025.06 | 4.28 | 11.44 | 1,337 | |
| standard RAGReader Model=Llama3, Retrieved Documents=100, Compression=None2025.06 | 0.25 | 4.48 | 14,147 | |
| standard RAGReader model=Gemma2, Number of retrieved documents=1002025.06 | — | — | 14,075 |