Open Domain Question Answering on 2WikiMultihopQA
59.3EMDecomp
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| Decomp2022.12 | 59.3 | — | |
| IRCoT QABackbone=GPT3 (175B)2022.12 | 57.7 | 68 | |
| CORE (3B)Downstream LLM=Qwen2.5-14B-Instruct, Compression setting=Compression of top 10 docs (with compressor trained on top 5 docs), # tok=482025.08 | 40.71 | 46.33 | |
| SelfAsk2022.12 | 40.1 | — | |
| CORE (3B)Downstream LLM=Qwen2.5-14B-Instruct, Compression setting=Compression of top 5 docs, # tok=402025.08 | 39.05 | 44.74 | |
| RECOMP (3B)Downstream LLM=Qwen2.5-14B-Instruct, Compression setting=Compression of top 10 docs (with compressor trained on top 5 docs), # tok=502025.08 | 33.12 | 38.69 | |
| RECOMP (3B)Downstream LLM=Qwen2.5-14B-Instruct, Compression setting=Compression of top 5 docs, # tok=422025.08 | 32.74 | 38.06 | |
| Top10 DocumentsDownstream LLM=Qwen2.5-14B-Instruct, Compression setting=None, Retrieval setting=Top10, # tok=15312025.08 | 31.04 | 36.75 | |
| Top5 DocumentsDownstream LLM=Qwen2.5-14B-Instruct, Compression setting=None, Retrieval setting=Top5, # tok=7662025.08 | 29.64 | 35.21 | |
| Deepseek-V3 (671B)Downstream LLM=Qwen2.5-14B-Instruct, Compression setting=Compression of top 10 docs (with compressor trained on top 5 docs), # tok=402025.08 | 29 | 34.64 | |
| Top3 DocumentsDownstream LLM=Qwen2.5-14B-Instruct, Compression setting=None, Retrieval setting=Top3, # tok=4602025.08 | 27.89 | 33.58 | |
| Top1 DocumentDownstream LLM=Qwen2.5-14B-Instruct, Compression setting=None, Retrieval setting=Top1, # tok=1532025.08 | 26.79 | 31.87 | |
| No RetrievalDownstream LLM=Qwen2.5-14B-Instruct, Compression setting=None, # tok=02025.08 | 26.11 | 29.51 | |
| Deepseek-V3 (671B)Downstream LLM=Qwen2.5-14B-Instruct, Compression setting=Compression of top 5 docs, # tok=452025.08 | 25.07 | 30.31 | |
| llama3.2-3BDownstream LLM=Qwen2.5-14B-Instruct, Compression setting=Compression of top 10 docs (with compressor trained on top 5 docs), # tok=592025.08 | 24.91 | 30.11 | |
| llama3.2-3BDownstream LLM=Qwen2.5-14B-Instruct, Compression setting=Compression of top 5 docs, # tok=602025.08 | 24.85 | 29.95 |