# Self-Consolidating Language Models: Continual Knowledge Incorporation from Context Source: [Self-Consolidating Language Models: Continual Knowledge Incorporation from Context]() Collection index: [llms.txt]() Data updated: 2026-05-14T05:00:13.174Z Authors: Zekun Wang, Anant Gupta, Zihan Dong, Christopher J. MacLellan Published: 2026-05-08T00:50:09.000Z ## Abstract Large language models (LLMs) increasingly receive information as streams of passages, conversations, and long-context workflows. While longer context windows expose more evidence, they do not ensure that useful information is preserved and reused. We study continual context consolidation: writing current context into model weights while limiting interference with previously consolidated information. We propose \\textbf{S}elf-\\textbf{Co}nsolidating \\textbf{L}anguage Models (SCoL), a post-training framework in which, given current context, an LLM learns to generate textual update instructions specifying which of its own Transformer layers should be updated. Because committed updates change the model that later generates future selections, we train SCoL with meta-reinforcement learning over an evolving model state. We instantiate SCoL with supervised QA rewards on SQuAD knowledge incorporation and intrinsic likelihood-based rewards for LongBench v2 long-context consolidation. Across both settings, SCoL improves acquisition and retention over prompting, summarization, batch test-time training, and sequential finetuning baselines. Analysis of learned selection patterns shows that SCoL encourages the LLM to generate sparse update locations that align with layers of high Fisher information, suggesting that the model learns to route plasticity toward loss-sensitive regions while limiting interference. Moreover, SCoL transfers from shorter meta-training streams to longer LongBench v2 streams at evaluation, suggesting that our framework supports scalable streaming consolidation. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Long-context consolidation on LongBench short: 16k–32k tokens v2](): Long-context consolidation; LongBench short: 16k–32k tokens v2; Accuracy: 42.3. - [Continual Knowledge Incorporation on SQuAD 100 passages (val)](): Continual Knowledge Incorporation; SQuAD 100 passages (val); Immediate Accuracy: 35.4. - [Long-context consolidation on LongBench long: 32k–64k tokens v2](): Long-context consolidation; LongBench long: 32k–64k tokens v2; Accuracy: 37. ## Original sources and implementation [Original paper]() [PDF]()