# VectorFit : Adaptive Singular & Bias Vector Fine-Tuning of Pre-trained Foundation Models

Source: [VectorFit : Adaptive Singular & Bias Vector Fine-Tuning of Pre-trained Foundation Models](<https://www.sota2.com/research/paper/68fa7ab31e209131cea84b2b>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-28T13:57:20.578Z

Authors: Suhas G Hegde, Shilpy Kaur, Aruna Tiwari

Published: 2025-03-25T10:36:27.000Z

## Abstract

Popular PEFT methods reduce trainable parameter count for fine-tuning by parameterizing new low-rank or sparse trainable weights in parallel to the frozen pre-trained weights $W$. However, these weights are trained from scratch, and there exists a performance gap between these methods and full fine-tuning, especially in low-budget settings. We introduce VectorFit, a new way of parameterization that efficiently utilizes the existing knowledge embedded in $W$ by adaptively training their singular vectors and biases. We show that utilizing the structural and transformational properties of $W$ in this way can lead to high-rank incremental weight matrices $\\Delta W$, comparable to that of full fine-tuning. VectorFit delivers superior results with 9$\\boldsymbol\\times$ fewer trainable parameters than the leading PEFT methods. Through comprehensive experiments across 19 datasets covering a wide range of language and vision tasks such as natural language understanding and generation, question answering, image classification, and image generation, we demonstrate that VectorFit surpasses baselines in terms of performance as a function of parameter-efficiency.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2503.19530>)

[PDF](<https://arxiv.org/pdf/2503.19530.pdf>)
