Question Answering on Natural Questions (NQ)
49.3AccuracyT5
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| T5Model Architecture=T5, Retrieval Context (IR)=Yes2020.05 | 49.3 | — | |
| UNIFIEDQAModel Architecture=T5, Retrieval Context (IR)=Yes2020.05 | 49.3 | — | |
| RAGSetting=Fine-tuned, QA Environment=Open-Domain2020.05 | 44.5 | — | |
| UnifiedQA-BARTModel Architecture=BART, Retrieval Context (IR)=Yes2020.05 | 44.5 | — | |
| DPR+BARTModel Architecture=BART, Retrieval Context (IR)=Yes2020.05 | 42.2 | — | |
| BART-largeModel Architecture=BART-large, Retrieval Context (IR)=Yes2020.05 | 42.1 | — | |
| PaLM-2-Lshots=1-shot2023.07 | 37.5 | — | |
| T5-11B+SSMSetting=Fine-tuned, QA Environment=Closed-Book, Auxiliary Training=SSM2020.05 | 36.6 | — | |
| T5-11BSetting=Fine-tuned, QA Environment=Closed-Book2020.05 | 34.5 | — | |
| LLAMA 2shots=1-shot, size=70B2023.07 | 33 | — | |
| GPT-3Learning Protocol=Few-Shot, QA Environment=Closed-Book2020.05 | 29.9 | — | |
| PaLMshots=1-shot2023.07 | 29.3 | — | |
| LLaMA 2 13BModality=Pretrained2023.10 | 29 | — | |
| Mistral 7BModality=Pretrained2023.10 | 28.8 | — | |
| LLAMA 2 7BModality=Pretrained2023.10 | 24.7 | — | |
| GPT-3Learning Protocol=One-Shot, QA Environment=Closed-Book2020.05 | 23 | — | |
| GPT-3number of parameters=175B2023.02 | 22.6 | — | |
| MOUE 12A2Act.=2.2B, VP.=11.3B, Adaptation Protocol=supervised fine-tuning, Base Model=JetMoE-8E2026.03 | 21.4 | — | |
| MOUE 10A2Act.=2.2B, VP.=9.7B, Adaptation Protocol=supervised fine-tuning, Base Model=JetMoE-8E2026.03 | 20.9 | — | |
| MOUE 16A2Act.=2.2B, VP.=14.6B, Adaptation Protocol=supervised fine-tuning, Base Model=JetMoE-8E2026.03 | 20.5 | — | |
| Vanilla MoEAct.=2.2B, VP.=8.0B, Adaptation Protocol=supervised fine-tuning, Base Model=JetMoE-8E2026.03 | 20.4 | — | |
| LLMBRACESBase Model=Llama2-7B, Rank (r)=16, Param.=2.1M, Param. (%)=0.03, Zero-shot=true2025.03 | 20.3 | — | |
| LLMBRACESBase Model=Llama3-8B, Rank (r)=16, Param.=2.1M, Param. (%)=0.03, Zero-shot=true2025.03 | 19.53 | — | |
| LLMBRACESBase Model=Llama3-8B, Rank (r)=32, Param.=4.2M, Param. (%)=0.05, Zero-shot=true2025.03 | 19.25 | — | |
| MOUE 76A8Act.=1.3B, VP.=8.1B, Adaptation Protocol=supervised fine-tuning, Base Model=OLMoE-64E2026.03 | 19.1 | — | |
| Vanilla MoEAct.=1.3B, VP.=6.9B, Adaptation Protocol=supervised fine-tuning, Base Model=OLMoE-64E2026.03 | 18.1 | — | |
| MOUE 68A8Act.=1.3B, VP.=7.3B, Adaptation Protocol=supervised fine-tuning, Base Model=OLMoE-64E2026.03 | 18.1 | — | |
| MOUE 72A8Act.=1.3B, VP.=7.7B, Adaptation Protocol=supervised fine-tuning, Base Model=OLMoE-64E2026.03 | 18 | — | |
| ToolformerAPI calls=enabled2023.02 | 17.7 | — | |
| LLMBRACESBase Model=Llama2-7B, Rank (r)=32, Param.=4.2M, Param. (%)=0.06, Zero-shot=true2025.03 | 16.62 | — | |
| GPT-3Learning Protocol=Zero-Shot, QA Environment=Closed-Book2020.05 | 14.6 | — | |
| GPT-J2023.02 | 12.8 | — | |
| ToolformerAPI calls=disabled2023.02 | 12.6 | — | |
| GPT-J + CCpre-training=CCNet2023.02 | 12.2 | — | |
| OPTnumber of parameters=66B2023.02 | 11.4 | — | |
| LoRABase Model=Llama3-8B, Rank (r)=16, Param.=6.8M, Param. (%)=0.08, Zero-shot=true2025.03 | 11.14 | — | |
| Code-Llama 7BModality=Finetuned2023.10 | 11 | — | |
| MOUE 72A8Act.=1.3B, VP.=7.7B, Adaptation Protocol=continued pre-training, Base Model=OLMoE-64E2026.03 | 10.7 | — | |
| Vanilla MoEAct.=1.3B, VP.=6.9B, Adaptation Protocol=continued pre-training, Base Model=OLMoE-64E2026.03 | 9.6 | — | |
| LLMBRACESBase Model=Qwen2.5-1.5B, Rank (r)=32, Param.=1.2M, Param. (%)=0.08, Zero-shot=true2025.03 | 9.58 | — | |
| LLMBRACESBase Model=Qwen2.5-1.5B, Rank (r)=16, Param.=0.6M, Param. (%)=0.04, Zero-shot=true2025.03 | 9.5 | — | |
| MOUE 68A8Act.=1.3B, VP.=7.3B, Adaptation Protocol=continued pre-training, Base Model=OLMoE-64E2026.03 | 9.3 | — | |
| LoRABase Model=Llama3-8B, Rank (r)=32, Param.=13.6M, Param. (%)=0.17, Zero-shot=true2025.03 | 9.11 | — | |
| LoRABase Model=Llama2-7B, Rank (r)=32, Param.=16.8M, Param. (%)=0.25, Zero-shot=true2025.03 | 8.67 | — | |
| LoRABase Model=Llama2-7B, Rank (r)=16, Param.=8.4M, Param. (%)=0.12, Zero-shot=true2025.03 | 8.25 | — | |
| MOUE 76A8Act.=1.3B, VP.=8.1B, Adaptation Protocol=continued pre-training, Base Model=OLMoE-64E2026.03 | 8 | — | |
| LoRABase Model=Qwen2.5-1.5B, Rank (r)=32, Param.=4.4M, Param. (%)=0.28, Zero-shot=true2025.03 | 5.29 | — | |
| LoRABase Model=Qwen2.5-1.5B, Rank (r)=16, Param.=2.2M, Param. (%)=0.14, Zero-shot=true2025.03 | 4.76 | — | |
| AutoRefine-BaseRetrieval Strategy=Agent, Backbone Model=Qwen2.5-3B2026.03 | — | 46.7 | |
| AutoRefine-InstructRetrieval Strategy=Agent, Backbone Model=Qwen2.5-3B2026.03 | — | 43.6 | |
| Direct GenerationRetrieval Strategy=None, Backbone Model=Qwen2.5-3B2026.03 | — | 10.6 | |
| EXSEARCH-BaseRetrieval Strategy=Agent, Backbone Model=Qwen2.5-3B2026.03 | — | 36.8 | |
| InForageRetrieval Strategy=Agent, Backbone Model=Qwen2.5-3B2026.03 | — | 42.1 | |
| IRCoTRetrieval Strategy=IRCoT, Backbone Model=Qwen2.5-3B2026.03 | — | 11.1 | |
| Naive RAGRetrieval Strategy=Naive RAG, Backbone Model=Qwen2.5-3B2026.03 | — | 34.8 | |
| O2-SearcherRetrieval Strategy=Agent, Backbone Model=Qwen2.5-3B2026.03 | — | 44.4 | |
| R1-BaseRetrieval Strategy=None, Backbone Model=Qwen2.5-3B2026.03 | — | 22.6 | |
| R1-InstructRetrieval Strategy=None, Backbone Model=Qwen2.5-3B2026.03 | — | 21 | |
| ReSearch-BaseRetrieval Strategy=Agent, Backbone Model=Qwen2.5-3B2026.03 | — | 42.7 | |
| ReSearch-InstructRetrieval Strategy=Agent, Backbone Model=Qwen2.5-3B2026.03 | — | 36.5 | |
| SAPO-3B-BaseRetrieval Strategy=Agent, Backbone Model=Qwen2.5-3B2026.03 | — | 47.4 | |
| SAPO-3B-InstructRetrieval Strategy=Agent, Backbone Model=Qwen2.5-3B2026.03 | — | 46.9 | |
| SE-SearchRetrieval Strategy=Agent, Backbone Model=Qwen2.5-3B2026.03 | — | 47.5 | |
| Search-o1Retrieval Strategy=Agent, Backbone Model=Qwen2.5-3B2026.03 | — | 23.8 | |
| Search-R1-BaseRetrieval Strategy=Agent, Backbone Model=Qwen2.5-3B2026.03 | — | 42.1 | |
| Search-R1-InstructRetrieval Strategy=Agent, Backbone Model=Qwen2.5-3B2026.03 | — | 39.7 | |
| SFTRetrieval Strategy=None, Backbone Model=Qwen2.5-3B2026.03 | — | 24.9 | |
| ZeroSearch-BaseRetrieval Strategy=Agent, Backbone Model=Qwen2.5-3B2026.03 | — | 43 |