# All4One: Symbiotic Neighbour Contrastive Learning via Self-Attention and Redundancy Reduction

Source: [All4One: Symbiotic Neighbour Contrastive Learning via Self-Attention and Redundancy Reduction](<https://www.sota2.com/research/paper/68fa721b1e209131cea54846>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T02:18:23.013Z

Authors: Imanol G. Estepa, Ignacio Saras\\'ua, Bhalaji Nagarajan, Petia Radeva

Published: 2023-03-16T15:51:59.000Z

## Abstract

Nearest neighbour based methods have proved to be one of the most successful self-supervised learning (SSL) approaches due to their high generalization capabilities. However, their computational efficiency decreases when more than one neighbour is used. In this paper, we propose a novel contrastive SSL approach, which we call All4One, that reduces the distance between neighbour representations using ''centroids'' created through a self-attention mechanism. We use a Centroid Contrasting objective along with single Neighbour Contrasting and Feature Contrasting objectives. Centroids help in learning contextual information from multiple neighbours whereas the neighbour contrast enables learning representations directly from the neighbours and the feature contrast allows learning representations unique to the features. This combination enables All4One to outperform popular instance discrimination approaches by more than 1% on linear classification evaluation for popular benchmark datasets and obtains state-of-the-art (SoTA) results. Finally, we show that All4One is robust towards embedding dimensionalities and augmentations, surpassing NNCLR and Barlow Twins by more than 5% on low dimensionality and weak augmentation settings. The source code would be made available soon.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Image Classification on ImageNet 1k (train) - Epoch Progression Metrics](<https://www.sota2.com/research/sota/image-classification-on-imagenet-1k-train-epoch-progression-metrics.md>): Image Classification; ImageNet 1k (train); Accuracy @ 100 Epochs: 65.4.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2303.09417>)

[PDF](<https://arxiv.org/pdf/2303.09417.pdf>)