# CorpusBrain++: A Continual Generative Pre-Training Framework for Knowledge-Intensive Language Tasks Source: [CorpusBrain++: A Continual Generative Pre-Training Framework for Knowledge-Intensive Language Tasks]() Collection index: [llms.txt]() Data updated: 2025-10-26T02:24:11.561Z Authors: Jiafeng Guo, Changjiang Zhou, Ruqing Zhang, Jiangui Chen, Maarten de Rijke, Yixing Fan, Xueqi Cheng Published: 2024-02-26T17:35:44.000Z ## Abstract Knowledge-intensive language tasks (KILTs) typically require retrieving relevant documents from trustworthy corpora, e.g., Wikipedia, to produce specific answers. Very recently, a pre-trained generative retrieval model for KILTs, named CorpusBrain, was proposed and reached new state-of-the-art retrieval performance. However, most existing research on KILTs, including CorpusBrain, has predominantly focused on a static document collection, overlooking the dynamic nature of real-world scenarios, where new documents are continuously being incorporated into the source corpus. To address this gap, it is crucial to explore the capability of retrieval models to effectively handle the dynamic retrieval scenario inherent in KILTs. In this work, we first introduce the continual document learning (CDL) task for KILTs and build a novel benchmark dataset named KILT++ based on the original KILT dataset for evaluation. Then, we conduct a comprehensive study over the use of pre-trained CorpusBrain on KILT++. Unlike the promising results in the stationary scenario, CorpusBrain is prone to catastrophic forgetting in the dynamic scenario, hence hampering the retrieval performance. To alleviate this issue, we propose CorpusBrain++, a continual generative pre-training framework. Empirical results demonstrate the significant effectiveness and remarkable efficiency of CorpusBrain++ in comparison to both traditional and generative IR methods. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Continual Generative Information Retrieval on MS MARCO](): Continual Generative Information Retrieval; MS Marco; Score D0: 63.79. - [Continual Generative Information Retrieval on Natural Questions](): Continual Generative Information Retrieval; Natural Questions; D0 Score: 70.58. ## Original sources and implementation [Original paper]() [PDF]()