# StableGrad: Backward Scale Control without Batch Normalization Source: [StableGrad: Backward Scale Control without Batch Normalization]() Collection index: [llms.txt]() Data updated: 2026-05-21T05:00:13.694Z Authors: Jose I. Mestre, Alberto Fern\\'andez-Hern\\'andez, Cristian P\\'erez-Corral, Manuel F. Dolz, Enrique S. Quintana-Ort\\'i Published: 2026-05-19T13:49:30.000Z ## Abstract Training very deep neural networks requires controlling the propagation of magnitudes across depth. Without such control, activations and gradients may vanish, explode, or enter unstable regimes that make optimization fail. Modern architectures often mitigate this problem through Batch Normalization, residual connections, or other normalization layers, which repeatedly re-scale or bypass intermediate representations. However, these mechanisms are not always appropriate. In Physics-Informed Neural Networks (PINNs), the network represents a continuous physical field and its input derivatives define the training objective, making batch-dependent normalization problematic because it can introduce non-local dependencies into the predicted field and its derivatives. We propose StableGrad, an optimizer-level scale-control mechanism that corrects layer-wise weight-gradient imbalances without modifying the forward model. Because the normalization is applied only after backpropagation and before the optimizer update, the network output, its derivatives, and the physical residual remain unchanged. We analyze the effective training dynamics induced by this rescaling and evaluate StableGrad on deep PINNs as the target application, with BatchNorm-free convolutional networks serving as a diagnostic stress test. On PINN benchmarks, StableGrad improves matched-depth solution accuracy and makes deeper models more reliable under standard optimization. On ResNet and EfficientNet architectures, where removing Batch Normalization normally leads to training collapse, StableGrad stabilizes optimization without introducing any other architectural change. These results show that optimizer-level control of weight-gradient scale can provide a practical alternative when forward normalization is unavailable or undesirable. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [PINN Loss Evaluation on Burgers Equation](): Solving Burgers' equation with PINNs; Burgers; Validation L2 Error: 0.011. - [PINN Performance on Helmholtz Equation](): Solving Helmholtz equation with PINNs; Helmholtz; Validation L2 Error: 0.0023. - [PINN Solving of Poisson's Equation on Poisson (Loss Metrics)](): Solving Poisson's equation with PINNs; Poisson; Validation L2 Error: 0.000026. ## Original sources and implementation [Original paper]() [PDF]()