# Continual Instruction Tuning for Large Multimodal Models Source: [Continual Instruction Tuning for Large Multimodal Models]() Collection index: [llms.txt]() Data updated: 2025-10-26T02:02:40.220Z Authors: Jinghan He, Haiyun Guo, Ming Tang, Jinqiao Wang Published: 2023-11-27T15:04:48.000Z ## Abstract Instruction tuning is now a widely adopted approach to aligning large multimodal models (LMMs) to follow human intent. It unifies the data format of vision-language tasks, enabling multi-task joint training. However, vision-language tasks are constantly being created in practice. Instead of always re-training LMMs when new tasks arrive, continual learning offers flexibility for models to continually and efficiently exploit the evolving data. This work aims to explore the following two questions: 1) Do LMMs still suffer from catastrophic forgetting in continual instruction tuning? 2) Are the existing three classes of continual learning methods still applicable to the continual instruction tuning of LMMs? An extensive study is conducted to address the above questions. First, we establish the first benchmark in this setting and reveal that catastrophic forgetting is still observed when continually instruction-tuning LMMs. However, the multi-task joint instruction tuning can facilitate the model's continual learning ability and mitigate forgetting. Second, we integrate and adapt classic continual learning methods to our context, demonstrating the efficacy of data replay and model expansion strategies across diverse scenarios. In contrast, regularization-based methods only perform well on models that have been jointly instruction-tuned on multiple tasks. Third, we delve into the correlation and forgetting dynamics between vision-language task pairs and propose task-similarity-informed regularization and model expansion methods for continual instruction tuning of LMMs. Experimental results show that our approach consistently boosts the model's performance. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Continual Visual Instruction Tuning on CVIT Single-type instruction](): Continual Visual Instruction Tuning; CVIT Single-type instruction; AP (Average Precision): 65.29. - [Knowledge Retention on 16-task Sequential Unlearning Retained Data Last](): Knowledge Retention; 16-task Sequential Unlearning Retained Data Last; Specificity: 75.08. - [Continual Visual Instruction Tuning on CVIT Five-type instruction](): Continual Visual Instruction Tuning; CVIT Five-type instruction; ScienceQA Score: 63.45. - [Knowledge Retention on Sequential Unlearning 16-task Retained Data Avg](): Knowledge Retention; Sequential Unlearning 16-task Retained Data Avg; Specificity: 77.65. - [Knowledge Unlearning on 16-task Sequential Unlearning Forgotten Data Last](): Knowledge Unlearning; 16-task Sequential Unlearning Forgotten Data Last; Context-aware Refusal Rate (CRR): 59.87. - [Knowledge Unlearning on 16-task Sequential Unlearning Forgotten Data Avg](): Knowledge Unlearning; 16-task Sequential Unlearning Forgotten Data Avg; CRR: 53.94. ## Original sources and implementation [Original paper]() [PDF]()