List your company
ResearchFormerly WizWand

Open Domain Question Answering on Natural Questions (NQ)

60.7Exact Match (EM)DeepSeek-v3
1.773617.071832.3747.6682Jul 2, 2020Jun 23, 2021Jun 15, 2022Jun 7, 2023May 29, 2024May 21, 2025May 13, 2026
Updated 5mo ago

Evaluation results

MethodLinks
DeepSeek-v3Category=Candidate Models2026.0360.7
DeepSeek-R1Category=Candidate Models2026.0360.3
Claude-Sonnet-4.5Category=Candidate Models2026.0359.4
FineRouterCategory=Routers2026.0359
Llama-3.3-70BCategory=Candidate Models2026.0357.6
IPRCategory=Routers2026.0357.6
kNNCategory=Routers2026.0356.2
Mistral-LargeCategory=Candidate Models2026.0356
Qwen3-235B-A22BCategory=Candidate Models2026.0355.1
RouteLLMCategory=Routers2026.0355
Llama-4-MaverickCategory=Candidate Models2026.0353.8
MLPCategory=Routers2026.0353.6
Fusion-in-DecoderModel scale=large2020.0751.4
FIDReader Type=Generative2020.0951.4
Mistral-SmallCategory=Candidate Models2026.0350.3
RouterDCCategory=Routers2026.0350.3
GPT-OSS-120BCategory=Candidate Models2026.0348.3
GraphRouterCategory=Routers2026.0348.3
Fusion-in-DecoderModel scale=base2020.0748.2
Claude-Haiku-4.5Category=Candidate Models2026.0346.6
GAR+Reader Type=Generative2020.0945.3
RAG2020.0744.5
RAGReader Type=Generative2020.0944.5
M3POBackbone=Qwen2.5-3B-Instruct, Strategy=Multi-Path Perception Policy Optimization, Context Retrieval=Top-3 documents2025.1244.1
GAR+Reader Type=Extractive2020.0943.8
SpanSeqGen2020.0742.5
SpanSeqGenReader Type=Generative2020.0942.2
GARReader Type=Extractive2020.0941.8
DPR2020.0741.5
DPRReader Type=Extractive2020.0941.5
M3POBackbone=Qwen2.5-1.5B-Instruct, Strategy=Multi-Path Perception Policy Optimization, Context Retrieval=Top-3 documents2025.1241.4
REALM2020.0740.4
REALMReader Type=Extractive2020.0940.4
Qwen3-32BCategory=Candidate Models2026.0339.5
GARReader Type=Generative2020.0938.1
GRPOBackbone=Qwen2.5-3B-Instruct, Strategy=GRPO, Context Retrieval=Top-3 documents2025.1238.1
HRPOBackbone=Qwen2.5-3B-Instruct, Strategy=Hybrid Reasoning Policy Optimization, Context Retrieval=Top-3 documents2025.1237.8
BM25 (ours)Reader Type=Extractive2020.0937.7
T52020.0736.6
T5Reader Type=Generative2020.0936.6
HRPOBackbone=Qwen2.5-1.5B-Instruct, Strategy=Hybrid Reasoning Policy Optimization, Context Retrieval=Top-3 documents2025.1236.4
PPOBackbone=Qwen2.5-3B-Instruct, Strategy=PPO, Context Retrieval=Top-3 documents2025.1235.6
BM25 (ours)Reader Type=Generative2020.0935.3
RAGBackbone=Qwen2.5-7B-Instruct, Strategy=Retrieval-Augmented Generation, Context Retrieval=Top-3 documents2025.1234.9
RAGBackbone=Qwen2.5-3B-Instruct, Strategy=Retrieval-Augmented Generation, Context Retrieval=Top-3 documents2025.1234.8
Graph Retriever2020.0734.7
Graph RetrieverReader Type=Extractive2020.0934.5
ORQAReader Type=Extractive2020.0933.3
PPOBackbone=Qwen2.5-1.5B-Instruct, Strategy=PPO, Context Retrieval=Top-3 documents2025.1232.7
Path RetrieverReader Type=Extractive2020.0932.6
Path Retriever2020.0731.7
ORQA2020.0731.3
GPT-3shot=few-shot2020.0729.9
GPT-3Reader Type=Generative2020.0929.9
GRPOBackbone=Qwen2.5-1.5B-Instruct, Strategy=GRPO, Context Retrieval=Top-3 documents2025.1229.3
Hard EM2020.0728.8
RAGBackbone=Qwen2.5-1.5B-Instruct, Strategy=Retrieval-Augmented Generation, Context Retrieval=Top-3 documents2025.1228.8
Hard EMReader Type=Extractive2020.0928.1
SFTBackbone=Qwen2.5-3B-Instruct, Strategy=Supervised Fine-Tuning, Context Retrieval=Top-3 documents2025.1224.9
IRCoTBackbone=Qwen2.5-7B-Instruct, Strategy=Interleaved retrieval with CoT, Context Retrieval=Top-3 documents2025.1222.4
FIXED_E=128Experts=1282026.0516.34
Search-o1Backbone=Qwen2.5-7B-Instruct, Strategy=Search-o1, Context Retrieval=Top-3 documents2025.1215.1
QABackbone=Qwen2.5-7B-Instruct, Strategy=Direct inference, Context Retrieval=Top-3 documents2025.1213.4
EMO (Stage 5)E=64→1282026.0513.35
FIXED_E=32Experts=322026.0511.86
MobileLLM-Flash 1.4BParameter Count=1.4B, Evaluation Protocol=5-shot2026.0311.83
Gemma3 1BParameter Count=1B, Evaluation Protocol=5-shot2026.039.48
SFTBackbone=Qwen2.5-1.5B-Instruct, Strategy=Supervised Fine-Tuning, Context Retrieval=Top-3 documents2025.129.4
FIXED_E=16Experts=162026.059.11
EMO (Stage 4)E=32→642026.058.64
LFM2 1.2BParameter Count=1.2B, Evaluation Protocol=5-shot2026.038.6
MobileLLM-Flash 650MParameter Count=650M, Evaluation Protocol=5-shot2026.037.56
EMO (Stage 3)E=16→322026.057.23
LFM2 700MParameter Count=700M, Evaluation Protocol=5-shot2026.036.9
MobileLLM-Flash 350MParameter Count=350M, Evaluation Protocol=5-shot2026.035.9
EMO (Stage 1)E=82026.055.65
Llama3.2 1BParameter Count=1B, Evaluation Protocol=5-shot2026.035.48
EMO (Stage 2)E=8→162026.055.24
LFM2 350MParameter Count=350M, Evaluation Protocol=5-shot2026.034.96
CoTBackbone=Qwen2.5-7B-Instruct, Strategy=Chain-of-Thought, Context Retrieval=Top-3 documents2025.124.8
Qwen3 0.6BParameter Count=0.6B, Evaluation Protocol=5-shot2026.034.32
Gemma3 270MParameter Count=270M, Evaluation Protocol=5-shot2026.034.04