# SMI: Efficient Self-Supervised Learning via Mutual-Information-Inspired Dependency Optimization

Source: [SMI: Efficient Self-Supervised Learning via Mutual-Information-Inspired Dependency Optimization](<https://www.sota2.com/research/paper/6a279e4a90966ade919bf950>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-06-10T05:01:35.776Z

Authors: Pritam Mishra, Coloma Ballester, Dimosthenis Karatzas

Published: 2026-06-06T20:50:04.000Z

## Abstract

Self-supervised learning (SSL) has achieved remarkable representation learning performance, but many existing methods rely on large batch sizes, memory banks, momentum encoders, or global synchronization mechanisms that substantially increase computational cost and training complexity. In this work, we propose Semantic Mutual Information (SMI), a lightweight self-supervised objective derived from a mutual-information-inspired dependency formulation under Gaussian assumptions. Unlike conventional correlation matching objectives that operate on high-dimensional feature correlation matrices, SMI performs optimization on a sample-level dependency matrix through a nonlinear transformation of pairwise correlations. This formulation induces distinct optimization dynamics that emphasize strongly dependent semantic pairs while maintaining representation diversity. Experimental results on ImageNet using a ResNet-50 backbone demonstrate that SMI achieves competitive linear evaluation performance relative to state-of-the-art SSL approaches while substantially reducing computational complexity. Across multiple low-resource benchmarks, SMI consistently improves transfer performance over Barlow Twins, particularly on fine-grained datasets. Furthermore, analyses of optimization dynamics and representation geometry suggest improved alignment--redundancy balance, greater feature diversity, and more spatially localized semantic representations. These results indicate that nonlinear dependency optimization provides an effective and computationally efficient alternative to conventional correlation-based self-supervised learning objectives.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Image Classification on ImageNet (val) (Top-1/Top-5 Accuracy, Throughput)](<https://www.sota2.com/research/sota/image-classification-on-imagenet-val-top-1-top-5-accuracy-throughput.md>): Image Classification; ImageNet (val); Top-1 Accuracy: 71.41.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2606.08332>)

[PDF](<https://arxiv.org/pdf/2606.08332.pdf>)
