# Degradation of Feature Space in Continual Learning

Source: [Degradation of Feature Space in Continual Learning](<https://www.sota2.com/research/paper/698969edc97a0b3e3b5dc40a>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-02-10T05:00:04.897Z

Authors: Chiara Lanza, Roberto Pereira, Marco Miozzo, Eduard Angelats, Paolo Dini

Published: 2026-02-06T10:26:34.000Z

## Abstract

Centralized training is the standard paradigm in deep learning, enabling models to learn from a unified dataset in a single location. In such setup, isotropic feature distributions naturally arise as a mean to support well-structured and generalizable representations. In contrast, continual learning operates on streaming and non-stationary data, and trains models incrementally, inherently facing the well-known plasticity-stability dilemma. In such settings, learning dynamics tends to yield increasingly anisotropic feature space. This arises a fundamental question: should isotropy be enforced to achieve a better balance between stability and plasticity, and thereby mitigate catastrophic forgetting? In this paper, we investigate whether promoting feature-space isotropy can enhance representation quality in continual learning. Through experiments using contrastive continual learning techniques on CIFAR-10 and CIFAR-100 data, we find that isotropic regularization fails to improve, and can in fact degrade, model accuracy in continual settings. Our results highlight essential differences in feature geometry between centralized and continual learning, suggesting that isotropy, while beneficial in centralized setups, may not constitute an appropriate inductive bias for non-stationary learning scenarios.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Image Classification on CIFAR-10 50 + 50 scenario (test)](<https://www.sota2.com/research/sota/image-classification-on-cifar-10-50-plus-50-scenario-test.md>): Image Classification; CIFAR-10 50 + 50 scenario (test); Accuracy: 86.94.

- [Image Classification on CIFAR-10 40 + 30 + 30 scenario (test)](<https://www.sota2.com/research/sota/image-classification-on-cifar-10-40-plus-30-plus-30-scenario-test.md>): Image Classification; CIFAR-10 40 + 30 + 30 scenario (test); Accuracy: 78.83.

- [Image Classification on CIFAR-10 20 x 5 scenario (test)](<https://www.sota2.com/research/sota/image-classification-on-cifar-10-20-x-5-scenario-test.md>): Image Classification; CIFAR-10 20 x 5 scenario (test); Accuracy: 72.14.

- [Image Classification on CIFAR-100 50 + 50 scenario (test)](<https://www.sota2.com/research/sota/image-classification-on-cifar-100-50-plus-50-scenario-test.md>): Image Classification; CIFAR-100 50 + 50 scenario (test); Accuracy: 58.79.

- [Image Classification on CIFAR-100 40 + 30 + 30 scenario (test)](<https://www.sota2.com/research/sota/image-classification-on-cifar-100-40-plus-30-plus-30-scenario-test.md>): Image Classification; CIFAR-100 40 + 30 + 30 scenario (test); Accuracy: 50.38.

- [Image Classification on CIFAR-100 20 x 5 scenario (test)](<https://www.sota2.com/research/sota/image-classification-on-cifar-100-20-x-5-scenario-test.md>): Image Classification; CIFAR-100 20 x 5 scenario (test); Accuracy: 44.81.

- [Image Classification on CIFAR-100 Centralized (test)](<https://www.sota2.com/research/sota/image-classification-on-cifar-100-centralized-test.md>): Image Classification; CIFAR-100 Centralized (test); Accuracy: 73.97.

- [Image Classification on CIFAR-10 Centralized (test)](<https://www.sota2.com/research/sota/image-classification-on-cifar-10-centralized-test.md>): Image Classification; CIFAR-10 Centralized (test); Accuracy: 94.93.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2602.06586>)

[PDF](<https://arxiv.org/pdf/2602.06586.pdf>)
