# Orthogonalising gradients to speed up neural network optimisation

Source: [Orthogonalising gradients to speed up neural network optimisation](<https://www.sota2.com/research/paper/68fa6bd81e209131cea30898>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-25T23:58:04.948Z

Authors: Mark Tuddenham, Adam Pr\\"ugel-Bennett, Jonathan Hare

Published: 2022-02-14T21:46:07.000Z

## Abstract

The optimisation of neural networks can be sped up by orthogonalising the gradients before the optimisation step, ensuring the diversification of the learned representations. We orthogonalise the gradients of the layer's components/filters with respect to each other to separate out the intermediate representations. Our method of orthogonalisation allows the weights to be used more flexibly, in contrast to restricting the weights to an orthogonalised sub-space. We tested this method on ImageNet and CIFAR-10 resulting in a large decrease in learning time, and also obtain a speed-up on the semi-supervised learning BarlowTwins. We obtain similar accuracy to SGD without fine-tuning and better accuracy for na\\"ively chosen hyper-parameters.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2202.07052>)

[PDF](<https://arxiv.org/pdf/2202.07052.pdf>)