CLIMB: Centroid-Based Hierarchical Memory for Online Continual Self-Supervised Learning

About

Online Continual Self-Supervised Learning (OCSSL) aims to learn representations from a continuous stream of unlabeled data, without knowledge of task boundaries and under memory constraints. Existing methods rely either on replay buffers that exploit latent space structure, or on regularization alone. We present CLIMB (Continual Learning with Intelligent Memory Bank), which combines both simultaneously. Our method introduces a hierarchical centroid-based memory, bounded in total number of stored images, combined with knowledge distillation on replayed examples to limit representation drift. The memory groups similar images into centroids, providing hard-to-discriminate examples for contrastive learning while covering the diversity of observed distributions. Experiments on Split CIFAR-100 and Split ImageNet-100, on standard benchmarks from the state-of-the-art as well as a new protocol with irregular task distributions show that CLIMB outperforms state-of-the-art OCSSL methods.

Julien Lefebvre, Stefan Duffner, Mathieu Lefort · 2026

Related benchmarks

Task	Dataset	Result
Class-incremental learning	CIFAR-100 20 tasks	Accuracy—	58
Class-incremental learning	ImageNet-100 20 tasks	Average Accuracy—	16
Class-incremental classification	CIFAR100 50 tasks	Accuracy43.15	14
Class-incremental classification	ImageNet-100 50 tasks	Cumulative Accuracy46.22	7
Class-incremental classification	ImageNet 100 tasks	CA44.58	7
Class-incremental classification	CIFAR-100 tasks	Class Accuracy (CA)38.6	7
Classification	CIFAR-100 Split (50 tasks)	CA38.93	6
Classification	Split ImageNet-100 (20 tasks)	Accuracy (CA)47.99	6
Classification	Split ImageNet-100 50 tasks	Classification Accuracy (CA)47.94	6
Classification	CIFAR-100 Split (20 tasks)	Classification Accuracy39	6