# Towards Theoretically Inspired Neural Initialization Optimization

Source: [Towards Theoretically Inspired Neural Initialization Optimization](<https://www.sota2.com/research/paper/68fa6d9e1e209131cea3adf3>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-01-04T05:22:58.552Z

Authors: Yibo Yang, Hong Wang, Haobo Yuan, Zhouchen Lin

Published: 2022-10-12T06:49:16.000Z

## Abstract

Automated machine learning has been widely explored to reduce human efforts in designing neural architectures and looking for proper hyperparameters. In the domain of neural initialization, however, similar automated techniques have rarely been studied. Most existing initialization methods are handcrafted and highly dependent on specific architectures. In this paper, we propose a differentiable quantity, named GradCosine, with theoretical insights to evaluate the initial state of a neural network. Specifically, GradCosine is the cosine similarity of sample-wise gradients with respect to the initialized parameters. By analyzing the sample-wise optimization landscape, we show that both the training and test performance of a network can be improved by maximizing GradCosine under gradient norm constraint. Based on this observation, we further propose the neural initialization optimization (NIO) algorithm. Generalized from the sample-wise analysis into the real batch setting, NIO is able to automatically look for a better initialization with negligible cost compared with the training time. With NIO, we improve the classification performance of a variety of neural architectures on CIFAR-10, CIFAR-100, and ImageNet. Moreover, we find that our method can even help to train large vision Transformer architecture without warmup.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Image Classification on ImageNet (Top-1 Accuracy and Training Time)](<https://www.sota2.com/research/sota/image-classification-on-imagenet-top-1-accuracy-and-training-time.md>): Image Classification; ImageNet; Top-1 Acc: 76.71.

- [Image Classification on CIFAR-10 (test)](<https://www.sota2.com/research/sota/image-classification-on-cifar-10-test.md>): Image Classification; CIFAR-10 (test); Accuracy: 97.5.

- [Image Classification on CIFAR-100 (test)](<https://www.sota2.com/research/sota/image-classification-on-cifar-100-test.md>): Image Classification; CIFAR-100 (test); Accuracy: 81.83.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2210.05956>)

[PDF](<https://arxiv.org/pdf/2210.05956.pdf>)