# Balancing Stability and Plasticity in Sequentially Trained Early-Exiting Neural Networks Source: [Balancing Stability and Plasticity in Sequentially Trained Early-Exiting Neural Networks]() Collection index: [llms.txt]() Data updated: 2026-05-09T05:00:04.795Z Authors: Alaa Zniber, Ouassim Karrakchou, Mounir Ghogho Published: 2026-05-06T18:36:32.000Z ## Abstract Early-exiting neural networks enable adaptive inference by allowing inputs to exit at intermediate classifiers, reducing computation for easy samples while maintaining high accuracy. In practice, exits can be trained sequentially by incrementally adding them to a shared backbone; however, this sequential training can cause newly introduced exits to interfere with previously learned ones, degrading the performance of earlier classifiers. We address this problem by retaining the knowledge embedded in existing exits while allowing new ones to specialize. We propose two alternative approaches that operate at different levels of the model. The first constrains learning by protecting parameters that are important for previously trained exits, while the second preserves the output distributions of earlier exits as the network adapts. These alternatives directly reflect the stability-plasticity trade-off studied in continual learning. Accordingly, we leverage \\textit{Elastic Weight Consolidation} to constrain critical weights and \\textit{Learning without Forgetting} to preserve output distributions. Experiments on standard benchmarks show that our approaches consistently improve early-exit performance, achieving higher accuracy over existing sequential training methods and significant performance speedups at low computational budgets. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Image Classification on CIFAR-100 (test) (Accuracy vs FLOPs Budget)](): Image Classification; CIFAR-100 (test); Top-1 Accuracy (25% FLOPs): 0.7071. ## Original sources and implementation [Original paper]() [PDF]()