Question Answering on Musique (EM, F1, Time, Rate)
22.92EMLatentRAG△
Evaluation Results
| Method | Links | ||||
|---|---|---|---|---|---|
| LatentRAG△Retrieval Model=jina-embeddings-v5-text-nano2026.05 | 22.92 | — | 0.54 | — | |
| LatentRAG♢Retrieval Model=jina-embeddings-v5-text-nano2026.05 | 22.26 | — | 0.632 | — | |
| AutoRefine△Retrieval Model=e5-base-v22026.05 | 21.8 | — | 5.583 | — | |
| LatentRAG△Retrieval Model=e5-base-v22026.05 | 20.94 | — | 0.548 | — | |
| All-pool PortfolioModel=Llama-3.1-70B-Instruct, k=4, l=32026.05 | 20.89 | — | — | — | |
| LatentRAG△Retrieval Model=Qwen3-Embedding-0.6B2026.05 | 20.73 | — | 0.639 | — | |
| Vendi-RAG Adaptive (T=20) (MPNet)Model=Llama-3.1-70B-Instruct, T=202026.05 | 20.6 | — | — | — | |
| LatentRAG△Retrieval Model=harrier-oss-v1-270m2026.05 | 20.56 | — | 0.579 | — | |
| AutoRefine△Retrieval Model=jina-embeddings-v5-text-nano2026.05 | 20.52 | — | 5.614 | — | |
| Search-R1♢Retrieval Model=jina-embeddings-v5-text-nano2026.05 | 20.4 | — | 6.177 | — | |
| SAPO-3B-InstructRetrieval Strategy=Agent, Backbone Model=Qwen2.5-3B2026.03 | 20.3 | — | — | — | |
| Search-R1♢Retrieval Model=e5-base-v22026.05 | 20.19 | — | 6.416 | — | |
| AutoRefine△Retrieval Model=harrier-oss-v1-270m2026.05 | 20.15 | — | 5.511 | — | |
| LatentRAG♢Retrieval Model=harrier-oss-v1-270m2026.05 | 19.9 | — | 0.708 | — | |
| Vendi-RAG Adaptive (T=10) (MPNet)Model=Llama-3.1-70B-Instruct, T=102026.05 | 19.9 | — | — | — | |
| LatentRAG♢Retrieval Model=Qwen3-Embedding-0.6B2026.05 | 19.82 | — | 0.73 | — | |
| LatentRAG△Retrieval Model=F2LLM-v2-330M2026.05 | 19.82 | — | 0.566 | — | |
| Search-R1♢Retrieval Model=Qwen3-Embedding-0.6B2026.05 | 19.61 | — | 6.846 | — | |
| SAPO-3B-BaseRetrieval Strategy=Agent, Backbone Model=Qwen2.5-3B2026.03 | 19.6 | — | — | — | |
| All-pool PortfolioModel=Gemma-3-27B-It, k=4, l=32026.05 | 19.49 | — | — | — | |
| AutoRefine△Retrieval Model=F2LLM-v2-330M2026.05 | 19.03 | — | 5.412 | — | |
| LatentRAG♢Retrieval Model=e5-base-v22026.05 | 18.87 | — | 0.656 | — | |
| AutoRefine△Retrieval Model=Qwen3-Embedding-0.6B2026.05 | 18.66 | — | 5.553 | — | |
| Search-R1♢Retrieval Model=harrier-oss-v1-270m2026.05 | 18.62 | — | 6.613 | — | |
| Search-R1♢Retrieval Model=F2LLM-v2-330M2026.05 | 18.54 | — | 6.69 | — | |
| LatentRAG♢Retrieval Model=F2LLM-v2-330M2026.05 | 18.37 | — | 0.667 | — | |
| SE-SearchRetrieval Strategy=Agent, Backbone Model=Qwen2.5-3B2026.03 | 18.3 | — | — | — | |
| All-pool PortfolioModel=Llama-3.1-70B-Instruct, k=4, l=22026.05 | 18.2 | — | — | — | |
| StepSearch-BaseRetrieval Strategy=Agent, Backbone Model=Qwen2.5-3B2026.03 | 18.1 | — | — | — | |
| CriticSearchRetrieval Strategy=Agent, Backbone Model=Qwen2.5-3B2026.03 | 18 | — | — | — | |
| R1-SearcherRetrieval Model=Qwen3-Embedding-0.6B2026.05 | 17.83 | — | 10.063 | — | |
| All-pool PortfolioModel=Gemma-3-27B-It, k=4, l=22026.05 | 17.25 | — | — | — | |
| InForageRetrieval Strategy=Agent, Backbone Model=Qwen2.5-3B2026.03 | 17.2 | — | — | — | |
| AutoRefine-InstructRetrieval Strategy=Agent, Backbone Model=Qwen2.5-3B2026.03 | 16.9 | — | — | — | |
| O2-SearcherRetrieval Strategy=Agent, Backbone Model=Qwen2.5-3B2026.03 | 16 | — | — | — | |
| AutoRefine-BaseRetrieval Strategy=Agent, Backbone Model=Qwen2.5-3B2026.03 | 15.7 | — | — | — | |
| Best Vendi RetrieverModel=Gemma-3-27B-It2026.05 | 14.27 | — | — | — | |
| Vendi-Portfolio (MPNet)Model=Llama-3.1-70B-Instruct, k=5, l=22026.05 | 14.27 | — | — | — | |
| Best DS RetrieverModel=Gemma-3-27B-It2026.05 | 13.94 | — | — | — | |
| Nearest-neighbor retrieval (MPNet)Model=Llama-3.1-70B-Instruct2026.05 | 13.94 | — | — | — | |
| EXSEARCH-BaseRetrieval Strategy=Agent, Backbone Model=Qwen2.5-3B2026.03 | 13.8 | — | — | — | |
| DeepRAGRetrieval Model=Qwen3-Embedding-0.6B2026.05 | 13.2 | — | 4.149 | — | |
| Vendi-RAG Adaptive (T=20) (MPNet)Model=Gemma-3-27B-It, T=202026.05 | 13.07 | — | — | — | |
| ZeroSearch-BaseRetrieval Strategy=Agent, Backbone Model=Qwen2.5-3B2026.03 | 13 | — | — | — | |
| Nearest-neighbor retrieval (MPNet)Model=Gemma-3-27B-It2026.05 | 12.91 | — | — | — | |
| Vendi-Portfolio (MPNet)Model=Gemma-3-27B-It, k=5, l=22026.05 | 12.62 | — | — | — | |
| Vendi-RAG Adaptive (T=10) (MPNet)Model=Gemma-3-27B-It, T=102026.05 | 12.53 | — | — | — | |
| Search-R1-InstructRetrieval Strategy=Agent, Backbone Model=Qwen2.5-3B2026.03 | 12.4 | — | — | — | |
| ZeroSearchRetrieval Model=Qwen3-Embedding-0.6B2026.05 | 11.75 | — | 5.027 | — | |
| Best GraphDense RetrieverModel=Gemma-3-27B-It2026.05 | 11.34 | — | — | — | |
| Best Vendi RetrieverModel=Llama-3.1-70B-Instruct2026.05 | 11.21 | — | — | — | |
| Best DS RetrieverModel=Llama-3.1-70B-Instruct2026.05 | 10.92 | — | — | — | |
| Search-o1Retrieval Model=Qwen3-Embedding-0.6B2026.05 | 10.8 | — | 6.928 | — | |
| LooCompReader=Llama-3.3-70B Instruct, Retrieved chunks=Top-202026.03 | 10.7 | 20.1 | 0.164 | 7 | |
| ReSearch-InstructRetrieval Strategy=Agent, Backbone Model=Qwen2.5-3B2026.03 | 9.5 | — | — | — | |
| Best GraphDense RetrieverModel=Llama-3.1-70B-Instruct2026.05 | 8.73 | — | — | — | |
| ReSearch-BaseRetrieval Strategy=Agent, Backbone Model=Qwen2.5-3B2026.03 | 7.4 | — | — | — | |
| Iter-RetGenRetrieval Model=Qwen3-Embedding-0.6B2026.05 | 7.16 | — | 2.513 | — | |
| IRCoTRetrieval Strategy=IRCoT, Backbone Model=Qwen2.5-3B2026.03 | 6.7 | — | — | — | |
| Search-R1-BaseRetrieval Strategy=Agent, Backbone Model=Qwen2.5-3B2026.03 | 6.6 | — | — | — | |
| No retrievalModel=Gemma-3-27B-It2026.05 | 6.08 | — | — | — | |
| R1-InstructRetrieval Strategy=None, Backbone Model=Qwen2.5-3B2026.03 | 6 | — | — | — | |
| No retrievalModel=Llama-3.1-70B-Instruct2026.05 | 5.88 | — | — | — | |
| LooCompReader=Llama-3.1-8B Instruct, Retrieved chunks=Top-52026.03 | 5.7 | 12.7 | 0.038 | 10.8 | |
| R1-BaseRetrieval Strategy=None, Backbone Model=Qwen2.5-3B2026.03 | 5.5 | — | — | — | |
| Search-o1Retrieval Strategy=Agent, Backbone Model=Qwen2.5-3B2026.03 | 5.4 | — | — | — | |
| Naive RAGRetrieval Model=Qwen3-Embedding-0.6B2026.05 | 4.96 | — | 0.422 | — | |
| Naive RAGRetrieval Strategy=Naive RAG, Backbone Model=Qwen2.5-3B2026.03 | 4.7 | — | — | — | |
| SFTRetrieval Strategy=None, Backbone Model=Qwen2.5-3B2026.03 | 4.4 | — | — | — | |
| Direct InferRetrieval Model=N/A2026.05 | 3.19 | — | 0.239 | — | |
| Direct GenerationRetrieval Strategy=None, Backbone Model=Qwen2.5-3B2026.03 | 2 | — | — | — | |
| GRPOBackbone=Qwen2.5-7B-Instruct2025.10 | — | 25 | — | — | |
| GSPOBackbone=Qwen2.5-7B-Instruct2025.10 | — | 25.4 | — | — | |
| IGPOBackbone=Qwen2.5-7B-Instruct2025.10 | — | 32.7 | — | — | |
| PPOBackbone=Qwen2.5-7B-Instruct2025.10 | — | 26.2 | — | — | |
| Reinforce++Backbone=Qwen2.5-7B-Instruct2025.10 | — | 23.7 | — | — | |
| RLOOBackbone=Qwen2.5-7B-Instruct2025.10 | — | 24.8 | — | — |