# Efficient Task Adaptation in Large Language Models via Selective Parameter Optimization

Source: [Efficient Task Adaptation in Large Language Models via Selective Parameter Optimization](<https://www.sota2.com/research/paper/69e704c27d0172140ec7eef4>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-04-22T05:21:28.944Z

Authors: Weijie Wan, Jiangjiang Zhao

Published: 2026-04-18T16:13:49.000Z

## Abstract

Large Language Models (LLMs) have demonstrated excellent performance in general language understanding, generation and other tasks. However, when fine-tuning for specific domain tasks, the general knowledge accumulated in the pre-training phase is often partially overwritten or forgotten due to parameter updates, which severely limits the generalization ability and transferability of LLMs. Traditional fine-tuning strategies mostly train on the entire parameter space, ignoring the heterogeneity of model parameters, that is, some parameters are extremely important for general tasks, while other parameters are more sensitive to specific tasks. To alleviate the above problems, this paper innovatively proposes a parameter element importance evaluation method, which divides parameters into "core parameters" and "non-core parameters" by distinguishing the importance of parameters for general language ability tasks and specific domain tasks, and fixes the core parameters during fine-tuning, and only fine-tunes the non-core parameters. Extensive experiments on scientific, medical and physical tasks using GPT-J and LLaMA-3 show that our method can mitigate catastrophic forgetting while enhancing the adaptability of the model.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Scientific Reasoning on SciQ](<https://www.sota2.com/research/sota/scientific-reasoning-on-sciq.md>): Scientific Reasoning; SciQ; Accuracy: 97.08.

- [Physical Reasoning on PIQA (PPL, Acc)](<https://www.sota2.com/research/sota/physical-reasoning-on-piqa-ppl-acc.md>): Physical reasoning and problem-solving; PIQA; Perplexity: 3.38.

- [Medical information extraction and understanding on MedMCQA](<https://www.sota2.com/research/sota/medical-information-extraction-and-understanding-on-medmcqa.md>): Medical information extraction and understanding; MedMCQA; Perplexity (PPL): 3.36.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2604.17051>)

[PDF](<https://arxiv.org/pdf/2604.17051.pdf>)