# NG+ : A Multi-Step Matrix-Product Natural Gradient Method for Deep Learning

Source: [NG+ : A Multi-Step Matrix-Product Natural Gradient Method for Deep Learning](<https://www.sota2.com/research/paper/68fa58351e209131ce9ac37b>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-25T15:11:38.006Z

Authors: Minghan Yang, Dong Xu, Qiwen Cui, Zaiwen Wen, Pengxiang Xu

Published: 2021-06-14T14:31:31.000Z

## Abstract

In this paper, a novel second-order method called NG+ is proposed. By following the rule \`\`the shape of the gradient equals the shape of the parameter", we define a generalized fisher information matrix (GFIM) using the products of gradients in the matrix form rather than the traditional vectorization. Then, our generalized natural gradient direction is simply the inverse of the GFIM multiplies the gradient in the matrix form. Moreover, the GFIM and its inverse keeps the same for multiple steps so that the computational cost can be controlled and is comparable with the first-order methods. A global convergence is established under some mild conditions and a regret bound is also given for the online learning setting. Numerical results on image classification with ResNet50, quantum chemistry modeling with Schnet, neural machine translation with Transformer and recommendation system with DLRM illustrate that GN+ is competitive with the state-of-the-art methods.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2106.07454>)

[PDF](<https://arxiv.org/pdf/2106.07454.pdf>)
