# CCQA: A New Web-Scale Question Answering Dataset for Model Pre-Training Source: [CCQA: A New Web-Scale Question Answering Dataset for Model Pre-Training]() Collection index: [llms.txt]() Data updated: 2025-10-26T00:12:46.566Z Authors: Patrick Huber, Armen Aghajanyan, Barlas O\\u{g}uz, Dmytro Okhonko, Wen-tau Yih, Sonal Gupta, Xilun Chen Published: 2021-10-14T21:23:01.000Z ## Abstract With the rise of large-scale pre-trained language models, open-domain question-answering (ODQA) has become an important research topic in NLP. Based on the popular pre-training fine-tuning approach, we posit that an additional in-domain pre-training stage using a large-scale, natural, and diverse question-answering (QA) dataset can be beneficial for ODQA. Consequently, we propose a novel QA dataset based on the Common Crawl project in this paper. Using the readily available schema.org annotation, we extract around 130 million multilingual question-answer pairs, including about 60 million English data-points. With this previously unseen number of natural QA pairs, we pre-train popular language models to show the potential of large-scale in-domain pre-training for the task of question-answering. In our experiments, we find that pre-training question-answering models on our Common Crawl Question Answering dataset (CCQA) achieves promising results in zero-shot, low resource and fine-tuned settings across multiple tasks, models and benchmarks. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. ## Original sources and implementation [Original paper]() [PDF]()