# Hamiltonian Deep Neural Networks Guaranteeing Non-vanishing Gradients by Design

Source: [Hamiltonian Deep Neural Networks Guaranteeing Non-vanishing Gradients by Design](<https://www.sota2.com/research/paper/68fa6e4a1e209131cea3f050>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T00:54:03.791Z

Authors: Clara Luc\\'ia Galimberti, Luca Furieri, Liang Xu, Giancarlo Ferrari-Trecate

Published: 2021-05-27T14:52:22.000Z

## Abstract

Deep Neural Networks (DNNs) training can be difficult due to vanishing and exploding gradients during weight optimization through backpropagation. To address this problem, we propose a general class of Hamiltonian DNNs (H-DNNs) that stem from the discretization of continuous-time Hamiltonian systems and include several existing DNN architectures based on ordinary differential equations. Our main result is that a broad set of H-DNNs ensures non-vanishing gradients by design for an arbitrary network depth. This is obtained by proving that, using a semi-implicit Euler discretization scheme, the backward sensitivity matrices involved in gradient computations are symplectic. We also provide an upper-bound to the magnitude of sensitivity matrices and show that exploding gradients can be controlled through regularization. Finally, we enable distributed implementations of backward and forward propagation algorithms in H-DNNs by characterizing appropriate sparsity constraints on the weight matrices. The good performance of H-DNNs is demonstrated on benchmark classification problems, including image classification with the MNIST dataset.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2105.13205>)

[PDF](<https://arxiv.org/pdf/2105.13205.pdf>)
