# DSI++: Updating Transformer Memory with New Documents Source: [DSI++: Updating Transformer Memory with New Documents]() Collection index: [llms.txt]() Data updated: 2025-10-26T02:06:39.354Z Authors: Sanket Vaibhav Mehta, Jai Gupta, Yi Tay, Mostafa Dehghani, Vinh Q. Tran, Jinfeng Rao, Marc Najork, Emma Strubell, Donald Metzler Published: 2022-12-19T18:59:34.000Z ## Abstract Differentiable Search Indices (DSIs) encode a corpus of documents in model parameters and use the same model to answer user queries directly. Despite the strong performance of DSI models, deploying them in situations where the corpus changes over time is computationally expensive because reindexing the corpus requires re-training the model. In this work, we introduce DSI++, a continual learning challenge for DSI to incrementally index new documents while being able to answer queries related to both previously and newly indexed documents. Across different model scales and document identifier representations, we show that continual indexing of new documents leads to considerable forgetting of previously indexed documents. We also hypothesize and verify that the model experiences forgetting events during training, leading to unstable learning. To mitigate these issues, we investigate two approaches. The first focuses on modifying the training dynamics. Flatter minima implicitly alleviate forgetting, so we optimize for flatter loss basins and show that the model stably memorizes more documents ($+12\\%$). Next, we introduce a generative memory to sample pseudo-queries for documents and supplement them during continual indexing to prevent forgetting for the retrieval task. Extensive experiments on novel continual indexing benchmarks based on Natural Questions (NQ) and MS MARCO demonstrate that our proposed solution mitigates forgetting significantly. Concretely, it improves the average Hits@10 by $+21.1\\%$ over competitive baselines for NQ and requires $6$ times fewer model updates compared to re-training the DSI model for incrementally indexing five corpora in a sequence. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Dynamic Corpus Retrieval on MS MARCO D\_new](): Dynamic Corpus Retrieval; MS MARCO D\_new; Hits@1: 67.6. - [Generative Retrieval on NQ initial documents](): Generative Retrieval; NQ initial documents; R@1: 69.6. - [Generative Retrieval on MS-MARCO initial documents](): Generative Retrieval; MS-MARCO initial documents; R@10: 0.775. - [Generative Retrieval on NQ new documents](): Generative Retrieval; NQ new documents; R@1: 67.4. - [Generative Retrieval on MS-MARCO new documents](): Generative Retrieval; MS-MARCO new documents; R@10: 0.759. - [Dynamic Corpus Retrieval on NQ320K D\_new](): Dynamic Corpus Retrieval; NQ320K D\_new; Hits@1: 60.4. - [Retrieval on NQ (initial documents)](): Retrieval; NQ initial documents; R@1: 67.4. - [Dynamic Corpus Retrieval on MS MARCO D (train)](): Dynamic Corpus Retrieval; MS MARCO D (train); Hits@1: 48.5. - [Dynamic Corpus Retrieval on NQ320K D (train)](): Dynamic Corpus Retrieval; NQ320K D (train); Hits@1: 0.422. - [Continual Generative Information Retrieval on MS MARCO](): Continual Generative Information Retrieval; MS Marco; Score D0: 64.53. - [Continual Generative Information Retrieval on Natural Questions](): Continual Generative Information Retrieval; Natural Questions; D0 Score: 72.15. ## Original sources and implementation [Original paper]() [PDF]()