# Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasets

Source: [Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasets](<https://www.sota2.com/research/paper/68fa68101e209131cea19f6e>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-25T22:28:00.017Z

Authors: Patrick Lewis, Pontus Stenetorp, Sebastian Riedel

Published: 2020-08-06T13:17:43.000Z

## Abstract

Ideally Open-Domain Question Answering models should exhibit a number of competencies, ranging from simply memorizing questions seen at training time, to answering novel question formulations with answers seen during training, to generalizing to completely novel questions with novel answers. However, single aggregated test set scores do not show the full picture of what capabilities models truly have. In this work, we perform a detailed study of the test sets of three popular open-domain benchmark datasets with respect to these competencies. We find that 60-70% of test-time answers are also present somewhere in the training sets. We also find that 30% of test-set questions have a near-duplicate paraphrase in their corresponding training sets. Using these findings, we evaluate a variety of popular open-domain models to obtain greater insight into what extent they can actually generalize, and what drives their overall performance. We find that all models perform dramatically worse on questions that cannot be memorized from training sets, with a mean absolute performance difference of 63% between repeated and non-repeated data. Finally we show that simple nearest-neighbor models out-perform a BART closed-book QA model, further highlighting the role that training set memorization plays in these benchmarks

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Open-Domain Question Answering on WebQuestions (WQ)](<https://www.sota2.com/research/sota/open-domain-question-answering-on-webquestions-wq.md>): Open-domain Question Answering; WebQuestions (WQ); Exact Match (EM): 28.9.

- [Open-Domain Question Answering on Natural Questions (NQ) (EM, Q/s)](<https://www.sota2.com/research/sota/open-domain-question-answering-on-natural-questions-nq-em-q-s.md>): Open-domain Question Answering; NQ (Natural Questions); EM: 26.7.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2008.02637>)

[PDF](<https://arxiv.org/pdf/2008.02637.pdf>)