# Feature Starvation as Geometric Instability in Sparse Autoencoders Source: [Feature Starvation as Geometric Instability in Sparse Autoencoders]() Collection index: [llms.txt]() Data updated: 2026-05-09T05:00:04.795Z Authors: Faris Chaudhry, Keisuke Yano, Anthea Monod Published: 2026-05-06T18:11:14.000Z ## Abstract Sparse autoencoders (SAEs) are used to disentangle the dense, polysemantic internal representations of large language models (LLMs) into interpretable, monosemantic concepts. However, standard $\\ell\_1$-regularized SAEs suffer from feature starvation (dead neurons) and shrinkage bias, often requiring computationally expensive heuristic resampling and nondifferentiable hard-masking methods to bypass these challenges. We argue that feature starvation is not merely an empirical artifact of poor data diversity, but a fundamental optimization-geometric pathology of overcomplete dictionaries: the $\\ell\_1$-induced sparse coding map is unstable and fundamentally misaligned with shallow, amortized encoders. To address this structural instability, we introduce adaptive elastic net SAEs (AEN-SAEs), a fully differentiable architecture grounded in classical sparse regression. AEN-SAEs combine an $\\ell\_2$ structural term that enforces strong convexity and Lipschitz stability with adaptive $\\ell\_1$ reweighting that eliminates shrinkage bias and suppresses spurious features, thereby jointly controlling the curvature and interaction structure of the induced polyhedral geometry. Theoretically, we show that AEN-SAEs yield a Lipschitz-continuous sparse coding map and recover the global feature support under mild assumptions. Empirically, across synthetic settings and LLMs (Pythia 70M, Llama 3.1 8B), AEN-SAEs mitigate feature starvation without auxiliary heuristics while maintaining competitive reconstruction abilities. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Sparse Autoencoding on Pythia 70M (layer-3 residuals)](): Sparse Autoencoding; Pythia 70M (layer-3 residuals); Achieved L0: 33.9. - [Sparse Representation Learning on Llama 3.1 8B activations](): Sparse representation learning; Llama 3.1 8B activations; Achieved L0: 125.8. ## Original sources and implementation [Original paper]() [PDF]()