End-to-end Question Answering on HotpotQA (test val)
36.68EMSetR-Selection only
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| SetR-Selection onlyRetrieval (First-stage)=bge-large-en-v1.5, Ranking Strategy=SET SELECTION, # of Passages=3.412025.07 | 36.68 | 37.84 | |
| SetR-CoT & IRIRetrieval (First-stage)=bge-large-en-v1.5, Ranking Strategy=SET SELECTION, # of Passages=2.912025.07 | 36.62 | 38.11 | |
| SetR-CoTRetrieval (First-stage)=bge-large-en-v1.5, Ranking Strategy=SET SELECTION, # of Passages=2.882025.07 | 36.46 | 38.2 | |
| RankGPT (gpt-4o)Retrieval (First-stage)=bge-large-en-v1.5, Ranking Strategy=RERANKING, # of Passages=52025.07 | 33.85 | 34.45 | |
| bge-reranker-largeRetrieval (First-stage)=bge-large-en-v1.5, Ranking Strategy=RERANKING, # of Passages=52025.07 | 32.48 | 33.24 | |
| SetR-CoT & IRIRetrieval (First-stage)=BM25, Ranking Strategy=SET SELECTION, # of Passages=2.632025.07 | 32.2 | 30.57 | |
| RankVicunaRetrieval (First-stage)=bge-large-en-v1.5, Ranking Strategy=RERANKING, # of Passages=52025.07 | 32.08 | 32.83 | |
| RankLlamaRetrieval (First-stage)=bge-large-en-v1.5, Ranking Strategy=RERANKING, # of Passages=52025.07 | 31.88 | 32.95 | |
| RankZephyrRetrieval (First-stage)=bge-large-en-v1.5, Ranking Strategy=RERANKING, # of Passages=52025.07 | 31.83 | 32.97 | |
| SetR-Selection onlyRetrieval (First-stage)=BM25, Ranking Strategy=SET SELECTION, # of Passages=2.952025.07 | 31.61 | 30.55 | |
| RankGPT (gpt-4o)Retrieval (First-stage)=BM25, Ranking Strategy=RERANKING, # of Passages=52025.07 | 30.89 | 29.24 | |
| SetR-CoTRetrieval (First-stage)=BM25, Ranking Strategy=SET SELECTION, # of Passages=2.482025.07 | 30.79 | 30.12 | |
| FirstMistralRetrieval (First-stage)=bge-large-en-v1.5, Ranking Strategy=RERANKING, # of Passages=52025.07 | 30.1 | 31.07 | |
| bge-large-en-v1.5Retrieval (First-stage)=bge-large-en-v1.5, Ranking Strategy=RETRIEVAL ONLY, # of Passages=52025.07 | 30.07 | 30.97 | |
| bge-reranker-largeRetrieval (First-stage)=BM25, Ranking Strategy=RERANKING, # of Passages=52025.07 | 29.71 | 28.08 | |
| RankLlamaRetrieval (First-stage)=BM25, Ranking Strategy=RERANKING, # of Passages=52025.07 | 29.48 | 27.82 | |
| RankZephyrRetrieval (First-stage)=BM25, Ranking Strategy=RERANKING, # of Passages=52025.07 | 28.96 | 27.76 | |
| RankVicunaRetrieval (First-stage)=BM25, Ranking Strategy=RERANKING, # of Passages=52025.07 | 28.69 | 27.31 | |
| BM25Retrieval (First-stage)=BM25, Ranking Strategy=RETRIEVAL ONLY, # of Passages=52025.07 | 26.9 | 25.86 | |
| FirstMistralRetrieval (First-stage)=BM25, Ranking Strategy=RERANKING, # of Passages=52025.07 | 26.71 | 26.1 |