List your company
ResearchFormerly WizWand

Question Answering on Natural Questions (NQ)

49.3AccuracyT5
2.978415.004227.0339.0558May 2, 2020Apr 22, 2021Apr 13, 2022Apr 3, 2023Mar 24, 2024Mar 14, 2025Mar 5, 2026
Updated 7mo ago

Evaluation results

MethodLinks
T5Model Architecture=T5, Retrieval Context (IR)=Yes2020.0549.3—
UNIFIEDQAModel Architecture=T5, Retrieval Context (IR)=Yes2020.0549.3—
RAGSetting=Fine-tuned, QA Environment=Open-Domain2020.0544.5—
UnifiedQA-BARTModel Architecture=BART, Retrieval Context (IR)=Yes2020.0544.5—
DPR+BARTModel Architecture=BART, Retrieval Context (IR)=Yes2020.0542.2—
BART-largeModel Architecture=BART-large, Retrieval Context (IR)=Yes2020.0542.1—
PaLM-2-Lshots=1-shot2023.0737.5—
T5-11B+SSMSetting=Fine-tuned, QA Environment=Closed-Book, Auxiliary Training=SSM2020.0536.6—
T5-11BSetting=Fine-tuned, QA Environment=Closed-Book2020.0534.5—
LLAMA 2shots=1-shot, size=70B2023.0733—
GPT-3Learning Protocol=Few-Shot, QA Environment=Closed-Book2020.0529.9—
PaLMshots=1-shot2023.0729.3—
LLaMA 2 13BModality=Pretrained2023.1029—
Mistral 7BModality=Pretrained2023.1028.8—
LLAMA 2 7BModality=Pretrained2023.1024.7—
GPT-3Learning Protocol=One-Shot, QA Environment=Closed-Book2020.0523—
GPT-3number of parameters=175B2023.0222.6—
MOUE 12A2Act.=2.2B, VP.=11.3B, Adaptation Protocol=supervised fine-tuning, Base Model=JetMoE-8E2026.0321.4—
MOUE 10A2Act.=2.2B, VP.=9.7B, Adaptation Protocol=supervised fine-tuning, Base Model=JetMoE-8E2026.0320.9—
MOUE 16A2Act.=2.2B, VP.=14.6B, Adaptation Protocol=supervised fine-tuning, Base Model=JetMoE-8E2026.0320.5—
Vanilla MoEAct.=2.2B, VP.=8.0B, Adaptation Protocol=supervised fine-tuning, Base Model=JetMoE-8E2026.0320.4—
LLMBRACESBase Model=Llama2-7B, Rank (r)=16, Param.=2.1M, Param. (%)=0.03, Zero-shot=true2025.0320.3—
LLMBRACESBase Model=Llama3-8B, Rank (r)=16, Param.=2.1M, Param. (%)=0.03, Zero-shot=true2025.0319.53—
LLMBRACESBase Model=Llama3-8B, Rank (r)=32, Param.=4.2M, Param. (%)=0.05, Zero-shot=true2025.0319.25—
MOUE 76A8Act.=1.3B, VP.=8.1B, Adaptation Protocol=supervised fine-tuning, Base Model=OLMoE-64E2026.0319.1—
Vanilla MoEAct.=1.3B, VP.=6.9B, Adaptation Protocol=supervised fine-tuning, Base Model=OLMoE-64E2026.0318.1—
MOUE 68A8Act.=1.3B, VP.=7.3B, Adaptation Protocol=supervised fine-tuning, Base Model=OLMoE-64E2026.0318.1—
MOUE 72A8Act.=1.3B, VP.=7.7B, Adaptation Protocol=supervised fine-tuning, Base Model=OLMoE-64E2026.0318—
ToolformerAPI calls=enabled2023.0217.7—
LLMBRACESBase Model=Llama2-7B, Rank (r)=32, Param.=4.2M, Param. (%)=0.06, Zero-shot=true2025.0316.62—
GPT-3Learning Protocol=Zero-Shot, QA Environment=Closed-Book2020.0514.6—
GPT-J2023.0212.8—
ToolformerAPI calls=disabled2023.0212.6—
GPT-J + CCpre-training=CCNet2023.0212.2—
OPTnumber of parameters=66B2023.0211.4—
LoRABase Model=Llama3-8B, Rank (r)=16, Param.=6.8M, Param. (%)=0.08, Zero-shot=true2025.0311.14—
Code-Llama 7BModality=Finetuned2023.1011—
MOUE 72A8Act.=1.3B, VP.=7.7B, Adaptation Protocol=continued pre-training, Base Model=OLMoE-64E2026.0310.7—
Vanilla MoEAct.=1.3B, VP.=6.9B, Adaptation Protocol=continued pre-training, Base Model=OLMoE-64E2026.039.6—
LLMBRACESBase Model=Qwen2.5-1.5B, Rank (r)=32, Param.=1.2M, Param. (%)=0.08, Zero-shot=true2025.039.58—
LLMBRACESBase Model=Qwen2.5-1.5B, Rank (r)=16, Param.=0.6M, Param. (%)=0.04, Zero-shot=true2025.039.5—
MOUE 68A8Act.=1.3B, VP.=7.3B, Adaptation Protocol=continued pre-training, Base Model=OLMoE-64E2026.039.3—
LoRABase Model=Llama3-8B, Rank (r)=32, Param.=13.6M, Param. (%)=0.17, Zero-shot=true2025.039.11—
LoRABase Model=Llama2-7B, Rank (r)=32, Param.=16.8M, Param. (%)=0.25, Zero-shot=true2025.038.67—
LoRABase Model=Llama2-7B, Rank (r)=16, Param.=8.4M, Param. (%)=0.12, Zero-shot=true2025.038.25—
MOUE 76A8Act.=1.3B, VP.=8.1B, Adaptation Protocol=continued pre-training, Base Model=OLMoE-64E2026.038—
LoRABase Model=Qwen2.5-1.5B, Rank (r)=32, Param.=4.4M, Param. (%)=0.28, Zero-shot=true2025.035.29—
LoRABase Model=Qwen2.5-1.5B, Rank (r)=16, Param.=2.2M, Param. (%)=0.14, Zero-shot=true2025.034.76—
AutoRefine-BaseRetrieval Strategy=Agent, Backbone Model=Qwen2.5-3B2026.03—46.7
AutoRefine-InstructRetrieval Strategy=Agent, Backbone Model=Qwen2.5-3B2026.03—43.6
Direct GenerationRetrieval Strategy=None, Backbone Model=Qwen2.5-3B2026.03—10.6
EXSEARCH-BaseRetrieval Strategy=Agent, Backbone Model=Qwen2.5-3B2026.03—36.8
InForageRetrieval Strategy=Agent, Backbone Model=Qwen2.5-3B2026.03—42.1
IRCoTRetrieval Strategy=IRCoT, Backbone Model=Qwen2.5-3B2026.03—11.1
Naive RAGRetrieval Strategy=Naive RAG, Backbone Model=Qwen2.5-3B2026.03—34.8
O2-SearcherRetrieval Strategy=Agent, Backbone Model=Qwen2.5-3B2026.03—44.4
R1-BaseRetrieval Strategy=None, Backbone Model=Qwen2.5-3B2026.03—22.6
R1-InstructRetrieval Strategy=None, Backbone Model=Qwen2.5-3B2026.03—21
ReSearch-BaseRetrieval Strategy=Agent, Backbone Model=Qwen2.5-3B2026.03—42.7
ReSearch-InstructRetrieval Strategy=Agent, Backbone Model=Qwen2.5-3B2026.03—36.5
SAPO-3B-BaseRetrieval Strategy=Agent, Backbone Model=Qwen2.5-3B2026.03—47.4
SAPO-3B-InstructRetrieval Strategy=Agent, Backbone Model=Qwen2.5-3B2026.03—46.9
SE-SearchRetrieval Strategy=Agent, Backbone Model=Qwen2.5-3B2026.03—47.5
Search-o1Retrieval Strategy=Agent, Backbone Model=Qwen2.5-3B2026.03—23.8
Search-R1-BaseRetrieval Strategy=Agent, Backbone Model=Qwen2.5-3B2026.03—42.1
Search-R1-InstructRetrieval Strategy=Agent, Backbone Model=Qwen2.5-3B2026.03—39.7
SFTRetrieval Strategy=None, Backbone Model=Qwen2.5-3B2026.03—24.9
ZeroSearch-BaseRetrieval Strategy=Agent, Backbone Model=Qwen2.5-3B2026.03—43