Question Answering on Natural Questions (test)
61.65EMBERT-base
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| BERT-basetraining_data=Natural Questions (human labels)2020.05 | 61.65 | 73.91 | — | — | — | — | — | |
| FiD + Distill.2021.12 | 54.7 | — | — | — | — | — | — | |
| EMDR22021.12 | 52.5 | — | — | — | — | — | — | |
| FiD (large)Pretrained model=T5-large, number of parameters=770M2021.06 | 51.4 | — | — | — | — | — | — | |
| FiD2021.12 | 51.4 | — | — | — | — | — | — | |
| BPRPretrained model=ELECTRA-large, number of parameters=335M, k=10002021.06 | 49 | — | — | — | — | — | — | |
| Info-HCVAEtraining_data=SQuAD (labeled) + NQ (synthetic), sampling_factor=N x 102020.05 | 48.44 | 62.69 | — | — | — | — | — | |
| FiD (base)Pretrained model=T5-base, number of parameters=220M2021.06 | 48.2 | — | — | — | — | — | — | |
| Info-HCVAEtraining_data=SQuAD (labeled) + NQ (synthetic), sampling_factor=N x 52020.05 | 48.19 | 62.21 | — | — | — | — | — | |
| Info-HCVAEtraining_data=SQuAD (labeled) + NQ (synthetic), sampling_factor=N x 32020.05 | 47.73 | 61.98 | — | — | — | — | — | |
| Info-HCVAEtraining_data=SQuAD (labeled) + NQ (synthetic), sampling_factor=N x 22020.05 | 46.95 | 61.34 | — | — | — | — | — | |
| Info-HCVAEtraining_data=SQuAD (labeled) + NQ (synthetic), sampling_factor=N x 12020.05 | 46.7 | 61.08 | — | — | — | — | — | |
| RETRO 7.5BRetrieval source=DPR retrieval2021.12 | 45.5 | — | — | — | — | — | — | |
| RAGPretrained model=BART-large, number of parameters=406M2021.06 | 44.5 | — | — | — | — | — | — | |
| RAG2021.12 | 44.5 | — | — | — | — | — | — | |
| xRAGComp. rate=x128, Retrieval status=w/ retrieval2026.01 | 44.13 | 3.39 | — | — | — | 34.04 | — | |
| StandardRAGComp. rate=-, Retrieval status=w/ retrieval2026.01 | 43.49 | 23.56 | — | — | — | 44.57 | — | |
| BERT-basetraining_data=SQuAD only (labeled)2020.05 | 42.77 | 57.29 | — | — | — | — | — | |
| BPRPretrained model=BERT-base, number of parameters=110M, k=10002021.06 | 41.6 | — | — | — | — | — | — | |
| DPR2021.12 | 41.5 | — | — | — | — | — | — | |
| Graph-R1 + EKABackbone=Qwen2.5-14B-Instruct2025.12 | 40.63 | 56.02 | — | — | — | — | — | |
| REALM2021.12 | 40.4 | — | — | — | — | — | — | |
| LLaMAModel Size=65B, Few-shot Setting=64-shot2023.02 | 39.9 | — | — | — | — | — | — | |
| PaLMModel Size=540B, Few-shot Setting=64-shot2023.02 | 39.6 | — | — | — | — | — | — | |
| CriSPOLLM=Claude Sonnet, shot_count=64-shot2024.10 | 38.7 | — | — | — | — | — | — | |
| CriSPOLLM=Claude Sonnet, shot_count=0-shot2024.10 | 38.3 | — | — | — | — | — | — | |
| CriSPOLLM=Claude Instant, shot_count=64-shot2024.10 | 37.8 | — | — | — | — | — | — | |
| LLMLingua-2Comp. rate=x3, Retrieval status=w/ retrieval2026.01 | 37.73 | 18.98 | — | — | — | 36.95 | — | |
| COCOMComp. rate=x4, Retrieval status=w/ retrieval2026.01 | 36.59 | 39.54 | — | — | — | 41.5 | — | |
| CriSPOLLM=Claude Instant, shot_count=0-shot2024.10 | 36.5 | — | — | — | — | — | — | |
| LLaMAModel Size=33B, Few-shot Setting=64-shot2023.02 | 36 | — | — | — | — | — | — | |
| ChinchillaModel Size=70B, Few-shot Setting=64-shot2023.02 | 35.5 | — | — | — | — | — | — | |
| COCOMComp. rate=x16, Retrieval status=w/ retrieval2026.01 | 35.18 | 41.04 | — | — | — | 41.94 | — | |
| LLaMAModel Size=65B, Few-shot Setting=5-shot2023.02 | 35 | — | — | — | — | — | — | |
| ManualLLM=Claude Instant, shot_count=0-shot2024.10 | 34 | — | — | — | — | — | — | |
| ManualLLM=Claude Instant, shot_count=64-shot2024.10 | 33.4 | — | — | — | — | — | — | |
| LLaMAModel Size=33B, Few-shot Setting=5-shot2023.02 | 32.9 | — | — | — | — | — | — | |
| ManualLLM=Claude Sonnet, shot_count=64-shot2024.10 | 32 | — | — | — | — | — | — | |
| LLaMAModel Size=13B, Few-shot Setting=64-shot2023.02 | 31.9 | — | — | — | — | — | — | |
| ArcAlignerComp. rate=x24, Retrieval status=w/ retrieval2026.01 | 31.72 | 37.82 | — | — | — | 37.04 | — | |
| ChinchillaModel Size=70B, Few-shot Setting=5-shot2023.02 | 31.5 | — | — | — | — | — | — | |
| LLaMAModel Size=65B, Few-shot Setting=1-shot2023.02 | 31 | — | — | — | — | — | — | |
| Baseline 7BEvaluation setting=closed book2021.12 | 30.4 | — | — | — | — | — | — | |
| GPT-3Model Size=175B, Few-shot Setting=64-shot2023.02 | 29.9 | — | — | — | — | — | — | |
| PaLMModel Size=540B, Few-shot Setting=1-shot2023.02 | 29.3 | — | — | — | — | — | — | |
| LLaMAModel Size=33B, Few-shot Setting=1-shot2023.02 | 28.3 | — | — | — | — | — | — | |
| GopherModel Size=280B, Few-shot Setting=64-shot2023.02 | 28.2 | — | — | — | — | — | — | |
| LLaMAModel Size=13B, Few-shot Setting=5-shot2023.02 | 28.1 | — | — | — | — | — | — | |
| PaLMModel Size=62B, Few-shot Setting=64-shot2023.02 | 27.6 | — | — | — | — | — | — | |
| NaiveComp. rate=-, Retrieval status=w/o retrieval2026.01 | 27.2 | 16.5 | — | — | — | 28.25 | — | |
| ManualLLM=Claude Sonnet, shot_count=0-shot2024.10 | 26.6 | — | — | — | — | — | — | |
| PaLMModel Size=62B, Few-shot Setting=1-shot2023.02 | 26.5 | — | — | — | — | — | — | |
| LLaMAModel Size=7B, Few-shot Setting=64-shot2023.02 | 26.1 | — | — | — | — | — | — | |
| LLaMAModel Size=33B, Few-shot Setting=0-shot2023.02 | 24.9 | — | — | — | — | — | — | |
| GopherModel Size=280B, Few-shot Setting=5-shot2023.02 | 24.5 | — | — | — | — | — | — | |
| LLaMAModel Size=65B, Few-shot Setting=0-shot2023.02 | 23.8 | — | — | — | — | — | — | |
| LLaMAModel Size=13B, Few-shot Setting=1-shot2023.02 | 23.4 | — | — | — | — | — | — | |
| GPT-3Model Size=175B, Few-shot Setting=1-shot2023.02 | 23 | — | — | — | — | — | — | |
| LLaMAModel Size=7B, Few-shot Setting=5-shot2023.02 | 22 | — | — | — | — | — | — | |
| PaLMModel Size=540B, Few-shot Setting=0-shot2023.02 | 21.2 | — | — | — | — | — | — | |
| LLaMAModel Size=13B, Few-shot Setting=0-shot2023.02 | 20.1 | — | — | — | — | — | — | |
| LLaMAModel Size=7B, Few-shot Setting=1-shot2023.02 | 18.7 | — | — | — | — | — | — | |
| PaLMModel Size=62B, Few-shot Setting=0-shot2023.02 | 18.1 | — | — | — | — | — | — | |
| LLaMAModel Size=7B, Few-shot Setting=0-shot2023.02 | 16.8 | — | — | — | — | — | — | |
| ChinchillaModel Size=70B, Few-shot Setting=0-shot2023.02 | 16.6 | — | — | — | — | — | — | |
| GPT-3Model Size=175B, Few-shot Setting=0-shot2023.02 | 14.6 | — | — | — | — | — | — | |
| PaLMModel Size=8B, Few-shot Setting=64-shot2023.02 | 14.6 | — | — | — | — | — | — | |
| PaLMModel Size=8B, Few-shot Setting=1-shot2023.02 | 10.6 | — | — | — | — | — | — | |
| GopherModel Size=280B, Few-shot Setting=0-shot2023.02 | 10.1 | — | — | — | — | — | — | |
| PaLMModel Size=8B, Few-shot Setting=0-shot2023.02 | 8.4 | — | — | — | — | — | — | |
| OPROLLM=Claude Instant2024.10 | 8 | — | — | — | — | — | — | |
| OPROLLM=Claude Sonnet2024.10 | 6.7 | — | — | — | — | — | — | |
| BART-largeEvaluation Mode=Zero-shot2022.03 | — | — | — | — | 10.44 | — | — | |
| BIGBIRD-ETCSetup=Fine-tuning2020.07 | — | — | 77.8 | 57.9 | — | — | — | |
| DPR+BARTEvaluation Mode=Zero-shot2022.03 | — | — | — | — | 16.37 | — | — | |
| FiDEvaluation Mode=Zero-shot2022.03 | — | — | — | — | 43.93 | — | — | |
| HybridRetriever=BERT, Generator=Qwen2.5-3B-Instruct2026.06 | — | 42.65 | — | — | — | — | 40.18 | |
| HybridRetriever=BGE-M3, Generator=Qwen2.5-3B-Instruct2026.06 | — | 49.27 | — | — | — | — | 47.27 | |
| HybridRetriever=Qwen3-Embedding-4B, Generator=Qwen2.5-3B-Instruct2026.06 | — | 48.03 | — | — | — | — | 46.21 | |
| LongLLMLinguaRetriever=BERT, Generator=Qwen2.5-3B-Instruct2026.06 | — | 23.04 | — | — | — | — | 20.8 | |
| LongLLMLinguaRetriever=BGE-M3, Generator=Qwen2.5-3B-Instruct2026.06 | — | 45.81 | — | — | — | — | 41.84 | |
| LongLLMLinguaRetriever=Qwen3-Embedding-4B, Generator=Qwen2.5-3B-Instruct2026.06 | — | 44.59 | — | — | — | — | 40.57 | |
| LongRAGRetriever=BERT, Generator=Qwen2.5-3B-Instruct2026.06 | — | 20.93 | — | — | — | — | 18.51 | |
| LongRAGRetriever=BGE-M3, Generator=Qwen2.5-3B-Instruct2026.06 | — | 45.98 | — | — | — | — | 42.19 | |
| LongRAGRetriever=Qwen3-Embedding-4B, Generator=Qwen2.5-3B-Instruct2026.06 | — | 47.44 | — | — | — | — | 43.29 | |
| MS-PoERetriever=BERT, Generator=Qwen2.5-3B-Instruct2026.06 | — | 22.39 | — | — | — | — | 19.74 | |
| MS-PoERetriever=BGE-M3, Generator=Qwen2.5-3B-Instruct2026.06 | — | 44.32 | — | — | — | — | 40.15 | |
| MS-PoERetriever=Qwen3-Embedding-4B, Generator=Qwen2.5-3B-Instruct2026.06 | — | 44.81 | — | — | — | — | 40.64 | |
| PINERetriever=BERT, Generator=Qwen2.5-3B-Instruct2026.06 | — | 19.15 | — | — | — | — | 18.43 | |
| PINERetriever=BGE-M3, Generator=Qwen2.5-3B-Instruct2026.06 | — | 45.47 | — | — | — | — | 44.2 | |
| PINERetriever=Qwen3-Embedding-4B, Generator=Qwen2.5-3B-Instruct2026.06 | — | 46.28 | — | — | — | — | 44.66 | |
| Qwen2.5-14BTraining Stage=Pre-RL2026.05 | — | — | — | — | — | 25.24 | — | |
| Qwen2.5-14BTraining Stage=Post-RL2026.05 | — | — | — | — | — | 41.88 | — | |
| Qwen2.5-32BTraining Stage=Pre-RL2026.05 | — | — | — | — | — | 27.94 | — | |
| Qwen2.5-32BTraining Stage=Post-RL2026.05 | — | — | — | — | — | 42.06 | — | |
| Qwen2.5-72BTraining Stage=Pre-RL2026.05 | — | — | — | — | — | 34.64 | — | |
| Qwen2.5-72BTraining Stage=Post-RL2026.05 | — | — | — | — | — | 49.61 | — | |
| Qwen2.5-7BTraining Stage=Pre-RL2026.05 | — | — | — | — | — | 21.42 | — | |
| Qwen2.5-7BTraining Stage=Post-RL2026.05 | — | — | — | — | — | 34.12 | — | |
| Qwen3-30B-A3BTraining Stage=Pre-RL2026.05 | — | — | — | — | — | 31.42 | — |