# Firefly Neural Architecture Descent: a General Approach for Growing Neural Networks

Source: [Firefly Neural Architecture Descent: a General Approach for Growing Neural Networks](<https://www.sota2.com/research/paper/68fa6a3c1e209131cea26e0f>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-25T23:19:37.191Z

Authors: Lemeng Wu, Bo Liu, Peter Stone, Qiang Liu

Published: 2021-02-17T04:47:18.000Z

## Abstract

We propose firefly neural architecture descent, a general framework for progressively and dynamically growing neural networks to jointly optimize the networks' parameters and architectures. Our method works in a steepest descent fashion, which iteratively finds the best network within a functional neighborhood of the original network that includes a diverse set of candidate network structures. By using Taylor approximation, the optimal network structure in the neighborhood can be found with a greedy selection procedure. We show that firefly descent can flexibly grow networks both wider and deeper, and can be applied to learn accurate but resource-efficient neural architectures that avoid catastrophic forgetting in continual learning. Empirically, firefly descent achieves promising results on both neural architecture search and continual learning. In particular, on a challenging continual image classification task, it learns networks that are smaller in size but have higher average accuracy than those learned by the state-of-the-art methods.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Image Classification on ImageNet-1K 1.0 (test) (Top-1 Accuracy)](<https://www.sota2.com/research/sota/image-classification-on-imagenet-1k-1-0-test-top-1-accuracy.md>): Image Classification; ImageNet-1k 1.0 (test); Top-1 Accuracy: 71.3.

- [Image Classification on CIFAR-100 (test)](<https://www.sota2.com/research/sota/image-classification-on-cifar-100-test.md>): Image Classification; CIFAR-100 (test); Accuracy: 77.22.

- [Image Classification on CIFAR-10 (test)](<https://www.sota2.com/research/sota/image-classification-on-cifar-10-test.md>): Image Classification; CIFAR-10 (test); Accuracy: 91.91.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2102.08574>)

[PDF](<https://arxiv.org/pdf/2102.08574.pdf>)
