# SaluNet: Enabling Total Plasticity in Normalization-Free Deep Networks

Source: [SaluNet: Enabling Total Plasticity in Normalization-Free Deep Networks](<https://www.sota2.com/research/paper/6a210682c10484bac23f00ad>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-06-04T05:00:05.280Z

Authors: Mourad Zaied

Published: 2026-06-01T22:09:06.000Z

## Abstract

Normalization layers such as BatchNorm and LayerNorm have long been considered essential for stable training in deep networks. This work demonstrates that they can be fully replaced by a single learnable activation mechanism. We identify a plasticity suppression effect induced by standard normalization: learnable activation parameters rapidly lose adaptability when paired with normalization layers. Motivated by this observation, we introduce SALU (Saturated Adaptive Linear Unit), \\\[ \\operatorname{SALU}(x;a,b) = \\frac{a x}{\\sqrt{1 + a b x^2}},\\quad a\>0,\\; b\>0 \\\] a bounded, learnable activation that provides intrinsic signal stabilization without relying on batch statistics or external affine parameters. Building on SALU, we propose SaluNet, a paradigm grounded in total plasticity: SALU replaces normalization layers, while SWALU and GALU replace standard activations. With ResNet-18, SaluNet-C-18 achieves 97.35\\% on CIFAR-10 and 83.25\\% on CIFAR-100 without normalization, maintaining 93.44\\% and 76.23\\% at batch size 1 where normalized architectures fail. For transformers, SaluNet-T improves over LayerNorm-GELU from 90.92\\% to 91.01\\% on CIFAR-10 and from 66.54\\% to 68.10\\% on CIFAR-100. SaluNet-C-50 reaches 78.67\\% Top-1 on ImageNet-1K at $224\\times224$, and $79.23\\%$ at $288\\times288$. These results suggest normalization layers suppress total plasticity, a property biological neurons inherently possess, enabling deep networks to learn effectively.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Image Classification on CIFAR-100 (test) (Top-1 and Top-5 Accuracy)](<https://www.sota2.com/research/sota/image-classification-on-cifar-100-test-top-1-and-top-5-accuracy.md>): Image Classification; CIFAR-100 (test); Top-1 Accuracy: 83.06.

- [Image Classification on CIFAR-10 (test) (Top-1 and Top-5 Accuracy)](<https://www.sota2.com/research/sota/image-classification-on-cifar-10-test-top-1-and-top-5-accuracy.md>): Image Classification; CIFAR-10 (test); Top-1 Accuracy: 97.18.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2606.02927>)

[PDF](<https://arxiv.org/pdf/2606.02927.pdf>)
