# Learning the Regularization Strength for Deep Fine-Tuning via a Data-Emphasized Variational Objective

Source: [Learning the Regularization Strength for Deep Fine-Tuning via a Data-Emphasized Variational Objective](<https://www.sota2.com/research/paper/68fa76ba1e209131cea6ec00>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T04:01:17.446Z

Authors: Ethan Harvey, Mikhail Petrov, Michael C. Hughes

Published: 2024-10-25T16:32:11.000Z

## Abstract

A number of popular transfer learning methods rely on grid search to select regularization hyperparameters that control over-fitting. This grid search requirement has several key disadvantages: the search is computationally expensive, requires carving out a validation set that reduces the size of available data for model training, and requires practitioners to specify candidate values. In this paper, we propose an alternative to grid search: directly learning regularization hyperparameters on the full training set via model selection techniques based on the evidence lower bound ("ELBo") objective from variational methods. For deep neural networks with millions of parameters, we specifically recommend a modified ELBo that upweights the influence of the data likelihood relative to the prior while remaining a valid bound on the evidence for Bayesian model selection. Our proposed technique overcomes all three disadvantages of grid search. We demonstrate effectiveness on image classification tasks on several datasets, yielding heldout accuracy comparable to existing approaches with far less compute time.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2410.19675>)

[PDF](<https://arxiv.org/pdf/2410.19675.pdf>)
