# Neuro-Inspired Deep Neural Networks with Sparse, Strong Activations

Source: [Neuro-Inspired Deep Neural Networks with Sparse, Strong Activations](<https://www.sota2.com/research/paper/68fa6cf21e209131cea36bce>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T00:21:36.840Z

Authors: Metehan Cekic, Can Bakiskan, Upamanyu Madhow

Published: 2022-02-26T06:19:05.000Z

## Abstract

While end-to-end training of Deep Neural Networks (DNNs) yields state of the art performance in an increasing array of applications, it does not provide insight into, or control over, the features being extracted. We report here on a promising neuro-inspired approach to DNNs with sparser and stronger activations. We use standard stochastic gradient training, supplementing the end-to-end discriminative cost function with layer-wise costs promoting Hebbian ("fire together," "wire together") updates for highly active neurons, and anti-Hebbian updates for the remaining neurons. Instead of batch norm, we use divisive normalization of activations (suppressing weak outputs using strong outputs), along with implicit $\\ell\_2$ normalization of neuronal weights. Experiments with standard image classification tasks on CIFAR-10 demonstrate that, relative to baseline end-to-end trained architectures, our proposed architecture (a) leads to sparser activations (with only a slight compromise on accuracy), (b) exhibits more robustness to noise (without being trained on noisy data), (c) exhibits more robustness to adversarial perturbations (without adversarial training).

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Image Classification on CIFAR-10 (test)](<https://www.sota2.com/research/sota/image-classification-on-cifar-10-test.md>): Image Classification; CIFAR-10 (test); Accuracy: 87.3.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2202.13074>)

[PDF](<https://arxiv.org/pdf/2202.13074.pdf>)
