Question Answering on Natural Question (NQ) (dev)
87.2F1Super-annotator
Evaluation results
| Method | Links | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Super-annotator2019.01 | 87.2 | — | — | — | — | — | — | — | 90 | 84.6 | |
| SGICBackbone=Llama2-7B-Chat2025.06 | 81.2 | 79 | — | — | — | — | — | — | — | — | |
| Llama2-7B-ChatTuning=LoRA Tuning2025.06 | 77.9 | 74.7 | — | — | — | — | — | — | — | — | |
| Super-annotator2019.01 | 75.7 | — | — | — | — | — | — | — | 79.1 | 72.6 | |
| Single Human2019.01 | 73.4 | — | — | — | — | — | — | — | 80.4 | 67.6 | |
| SGICBackbone=Phi-3.5-mini2025.06 | 67.5 | 65 | — | — | — | — | — | — | — | — | |
| LongT5Model Scale=xl, Input Length=8k, Attention=TGlobal2021.12 | 66.61 | 62.66 | — | — | — | — | — | — | — | — | |
| LongT5Model Scale=large, Input Length=4k, Attention=TGlobal2021.12 | 65.38 | 60.77 | — | — | — | — | — | — | — | — | |
| BERT jointpre-training=SQuAD 1.1 fine-tuned2019.01 | 64.7 | — | — | — | — | — | — | — | 61.3 | 68.4 | |
| LongT5Model Scale=large, Input Length=10k, Attention=Local2021.12 | 64.4 | 60.01 | — | — | — | — | — | — | — | — | |
| Phi-3.5-miniTuning=Full Tuning2025.06 | 64.3 | 58.7 | — | — | — | — | — | — | — | — | |
| FewshotQA w/ MINPROMPTNumber of training examples=128, Number of Parameters=406M2023.10 | 64.2 | — | — | — | — | — | — | — | — | — | |
| T5.1.1Model Scale=large, Input Length=3k2021.12 | 64.17 | 60.09 | — | — | — | — | — | — | — | — | |
| T5.1.1Model Scale=xl, Input Length=4k2021.12 | 64.07 | 60.75 | — | — | — | — | — | — | — | — | |
| LongT5Model Scale=large, Input Length=6k, Attention=TGlobal2021.12 | 63.38 | 59.17 | — | — | — | — | — | — | — | — | |
| FewshotQANumber of training examples=128, Number of Parameters=406M2023.10 | 63.3 | — | — | — | — | — | — | — | — | — | |
| LongT5Model Scale=base, Input Length=12k, Attention=TGlobal2021.12 | 62.44 | 58.12 | — | — | — | — | — | — | — | — | |
| LongT5Model Scale=large, Input Length=512, Attention=TGlobal2021.12 | 61.53 | 57.55 | — | — | — | — | — | — | — | — | |
| FewshotQANumber of training examples=64, Number of Parameters=406M2023.10 | 60.9 | — | — | — | — | — | — | — | — | — | |
| T5.1.1Model Scale=large, Input Length=5122021.12 | 60.68 | 57.29 | — | — | — | — | — | — | — | — | |
| FewshotQANumber of training examples=32, Number of Parameters=406M2023.10 | 60.6 | — | — | — | — | — | — | — | — | — | |
| FewshotQA w/ MINPROMPTNumber of training examples=64, Number of Parameters=406M2023.10 | 59.7 | — | — | — | — | — | — | — | — | — | |
| LongT5Model Scale=base, Input Length=36k, Attention=Local2021.12 | 59.66 | 55.77 | — | — | — | — | — | — | — | — | |
| RikiNet-RoBERTa2020.09 | 59.3 | — | — | — | — | — | — | — | — | — | |
| FewshotQA w/ MINPROMPTNumber of training examples=32, Number of Parameters=406M2023.10 | 59.3 | — | — | — | — | — | — | — | — | — | |
| LongT5Model Scale=base, Input Length=512, Attention=TGlobal2021.12 | 59.06 | 55.73 | — | — | — | — | — | — | — | — | |
| FewshotQA w/ MINPROMPTNumber of training examples=16, Number of Parameters=406M2023.10 | 58.5 | — | — | — | — | — | — | — | — | — | |
| LongT5Model Scale=base, Input Length=512, Attention=Local2021.12 | 58.24 | 54.39 | — | — | — | — | — | — | — | — | |
| LongT5Model Scale=large, Input Length=512, Attention=Local2021.12 | 58 | 55.19 | — | — | — | — | — | — | — | — | |
| Single Human2019.01 | 57.5 | — | — | — | — | — | — | — | 63.4 | 52.6 | |
| PMRNumber of training examples=128, Number of Parameters=406M2023.10 | 57.4 | — | — | — | — | — | — | — | — | — | |
| FewshotQANumber of training examples=16, Number of Parameters=406M2023.10 | 57.3 | — | — | — | — | — | — | — | — | — | |
| BERTwwm + SQUAD22020.09 | 57.2 | — | — | — | — | — | — | — | — | — | |
| Cluster-Formernumber of clusters=5122020.09 | 57.1 | — | — | — | — | — | — | — | — | — | |
| T5.1.1Model Scale=base, Input Length=6k2021.12 | 56.73 | 56.73 | — | — | — | — | — | — | — | — | |
| Cluster-Formernumber of clusters=642020.09 | 56.7 | — | — | — | — | — | — | — | — | — | |
| Cluster-Formernumber of clusters=2562020.09 | 56.7 | — | — | — | — | — | — | — | — | — | |
| Sliding Window2020.09 | 56.4 | — | — | — | — | — | — | — | — | — | |
| Locality-Sensitive Hashing2020.09 | 56.4 | — | — | — | — | — | — | — | — | — | |
| Sparse Attention2020.09 | 56.1 | — | — | — | — | — | — | — | — | — | |
| SGICBackbone=GPT-4o2025.06 | 55 | 65.2 | — | — | — | — | — | — | — | — | |
| DecAtt + DocReader2019.01 | 54.8 | — | — | — | — | — | — | — | 52.7 | 57 | |
| GPT-4oFine-tuning=None2025.06 | 53 | 63.3 | — | — | — | — | — | — | — | — | |
| BERT jointpre-training=SQuAD 1.1 fine-tuned2019.01 | 52.7 | — | — | — | — | — | — | — | 59.5 | 47.3 | |
| BERTjoint2020.09 | 52.7 | — | — | — | — | — | — | — | — | — | |
| T5.1.1Model Scale=base, Input Length=5122021.12 | 52.54 | 50.93 | — | — | — | — | — | — | — | — | |
| PMRNumber of training examples=64, Number of Parameters=406M2023.10 | 51.2 | — | — | — | — | — | — | — | — | — | |
| Splinter w/ MINPROMPTNumber of training examples=128, Number of Parameters=110M2023.10 | 51.2 | — | — | — | — | — | — | — | — | — | |
| SGICBackbone=GPT-4o-mini2025.06 | 48.8 | 64.4 | — | — | — | — | — | — | — | — | |
| PMRNumber of training examples=32, Number of Parameters=406M2023.10 | 48.5 | — | — | — | — | — | — | — | — | — | |
| GPT-4o-miniFine-tuning=None2025.06 | 47.5 | 62.9 | — | — | — | — | — | — | — | — | |
| SplinterNumber of training examples=128, Number of Parameters=110M2023.10 | 46.3 | — | — | — | — | — | — | — | — | — | |
| DocumentQA2019.01 | 46.1 | — | — | — | — | — | — | — | 47.5 | 44.7 | |
| Splinter w/ MINPROMPTNumber of training examples=64, Number of Parameters=110M2023.10 | 45.9 | — | — | — | — | — | — | — | — | — | |
| PMRNumber of training examples=16, Number of Parameters=406M2023.10 | 43.6 | — | — | — | — | — | — | — | — | — | |
| Splinter w/ MINPROMPTNumber of training examples=32, Number of Parameters=110M2023.10 | 42.8 | — | — | — | — | — | — | — | — | — | |
| SplinterNumber of training examples=64, Number of Parameters=110M2023.10 | 38.2 | — | — | — | — | — | — | — | — | — | |
| Splinter w/ MINPROMPTNumber of training examples=16, Number of Parameters=110M2023.10 | 37.6 | — | — | — | — | — | — | — | — | — | |
| SpanBERTNumber of training examples=128, Number of Parameters=110M2023.10 | 36 | — | — | — | — | — | — | — | — | — | |
| DocumentQA2019.01 | 35.7 | — | — | — | — | — | — | — | 38.6 | 33.2 | |
| DrQA2020.09 | 35.7 | — | — | — | — | — | — | — | — | — | |
| SplinterNumber of training examples=32, Number of Parameters=110M2023.10 | 33.6 | — | — | — | — | — | — | — | — | — | |
| DecAtt + DocReader2019.01 | 31.4 | — | — | — | — | — | — | — | 34.3 | 28.9 | |
| DecAtt + DocReader2020.09 | 31.4 | — | — | — | — | — | — | — | — | — | |
| RoBERTaNumber of training examples=128, Number of Parameters=110M2023.10 | 30.1 | — | — | — | — | — | — | — | — | — | |
| SpanBERTNumber of training examples=64, Number of Parameters=110M2023.10 | 29.7 | — | — | — | — | — | — | — | — | — | |
| SplinterNumber of training examples=16, Number of Parameters=110M2023.10 | 27.4 | — | — | — | — | — | — | — | — | — | |
| SpanBERTNumber of training examples=32, Number of Parameters=110M2023.10 | 25.1 | — | — | — | — | — | — | — | — | — | |
| RoBERTaNumber of training examples=64, Number of Parameters=110M2023.10 | 24.2 | — | — | — | — | — | — | — | — | — | |
| RoBERTaNumber of training examples=32, Number of Parameters=110M2023.10 | 22.9 | — | — | — | — | — | — | — | — | — | |
| SpanBERTNumber of training examples=16, Number of Parameters=110M2023.10 | 19.6 | — | — | — | — | — | — | — | — | — | |
| RoBERTaNumber of training examples=16, Number of Parameters=110M2023.10 | 17.3 | — | — | — | — | — | — | — | — | — | |
| BART-GST-A + FiD-GST-Areranker=BART-GST-A (Attention Projection), reader=FiD-GST-A (Attention Projection)2023.05 | — | 51.05 | — | — | — | — | — | — | — | — | |
| BART-GST-A + FiD-readerreranker=BART-GST-A (Attention Projection), reader=FiD-reader2023.05 | — | 50.68 | — | — | — | — | — | — | — | — | |
| BART-GST-M + FiD-GST-Mreranker=BART-GST-M (MLP Projection), reader=FiD-GST-M (MLP Projection)2023.05 | — | 51.4 | — | — | — | — | — | — | — | — | |
| BART-GST-M + FiD-readerreranker=BART-GST-M (MLP Projection), reader=FiD-reader2023.05 | — | 51.11 | — | — | — | — | — | — | — | — | |
| BART-reranker + FiD-GST-Areranker=BART-reranker, reader=FiD-GST-A (Attention Projection)2023.05 | — | 50.8 | — | — | — | — | — | — | — | — | |
| BART-reranker + FiD-GST-Mreranker=BART-reranker, reader=FiD-GST-M (MLP Projection)2023.05 | — | 50.76 | — | — | — | — | — | — | — | — | |
| BART-reranker + FiD-readerreranker=BART-reranker, reader=FiD-reader2023.05 | — | 50.33 | — | — | — | — | — | — | — | — | |
| BERT w/ 4M Synthetic Data AugmentationSource=Prior Work2019.09 | — | — | — | — | — | — | 55.1 | 65.9 | — | — | |
| BERT w/ SQuAD 1.1 PTSource=Prior Work2019.09 | — | — | — | — | — | — | 52.7 | 64.7 | — | — | |
| BERT_SSPTConfiguration=Pre-Training2019.09 | — | — | — | — | — | — | 54.83 | 66.75 | — | — | |
| BERT_WWMConfiguration=Pre-Training2019.09 | — | — | — | — | — | — | 55.35 | 66.04 | — | — | |
| BERT_WWM + SQuAD 2 PTConfiguration=Pre-Training2019.09 | — | — | — | — | — | — | 56.95 | 67.28 | — | — | |
| BERT_WWM + SQuAD 2 PT + AoAConfiguration=Pre-Training2019.09 | — | — | — | — | — | — | 57.22 | 68.24 | — | — | |
| BERT_WWM + SQuAD 2 PT + Layer ComboConfiguration=Pre-Training2019.09 | — | — | — | — | — | — | 57.15 | 67.08 | — | — | |
| BERT_WWM w/ 100K Similar Examples from MRC TasksConfiguration=Data Augmentation2019.09 | — | — | — | — | — | — | 54.68 | 65.82 | — | — | |
| BERT_WWM w/ 21K Random Examples from MRC TasksConfiguration=Data Augmentation2019.09 | — | — | — | — | — | — | 54.05 | 66.23 | — | — | |
| BERT_WWM w/ 21K Similar Examples from MRC TasksConfiguration=Data Augmentation2019.09 | — | — | — | — | — | — | 55.18 | 66.34 | — | — | |
| BERT_WWM w/ SOSConfiguration=Data Augmentation2019.09 | — | — | — | — | — | — | 55.81 | 66.67 | — | — | |
| BERT-baseInput length=512, #Params=110M2020.04 | — | — | — | — | — | — | 47.5 | 63.4 | — | — | |
| BERT-largeInput length=512, #Params=340M2020.04 | — | — | — | — | — | — | 52.7 | 64.7 | — | — | |
| BIGBIRD-ETCModel Size=Base2020.07 | — | — | — | — | — | — | 54.9 | 73.9 | — | — | |
| BIGBIRD-ITCModel Size=Base2020.07 | — | — | — | — | — | — | 53.3 | 70.8 | — | — | |
| DecAtt + Doc ReaderSource=Prior Work2019.09 | — | — | — | — | — | — | 31.4 | 54.8 | — | — | |
| Discrete Hard EM2019.09 | — | 28.8 | — | — | — | — | — | — | — | — | |
| ETCInput length=512, Configuration=shared, no CPC, no hard g2l, #Params=109M2020.04 | — | — | — | — | — | — | 47.8 | 64.5 | — | — | |
| ETCInput length=4096, Configuration=shared, no CPC, no hard g2l, #Params=109M2020.04 | — | — | — | — | — | — | 49.7 | 69.2 | — | — | |
| ETCInput length=4096, Configuration=fixed blocks, shared, no CPC, no hard g2l, #Params=109M2020.04 | — | — | — | — | — | — | 50.8 | 69.7 | — | — | |
| ETCInput length=4096, Configuration=shared, no hard g2l, #Params=109M2020.04 | — | — | — | — | — | — | 52.4 | 71.7 | — | — |