# How do Cross-View and Cross-Modal Alignment Affect Representations in Contrastive Learning?

Source: [How do Cross-View and Cross-Modal Alignment Affect Representations in Contrastive Learning?](<https://www.sota2.com/research/paper/68fa6e0a1e209131cea3d9a4>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T00:49:11.447Z

Authors: Thomas M. Hehn, Julian F.P. Kooij, Dariu M. Gavrila

Published: 2022-11-23T21:26:25.000Z

## Abstract

Various state-of-the-art self-supervised visual representation learning approaches take advantage of data from multiple sensors by aligning the feature representations across views and/or modalities. In this work, we investigate how aligning representations affects the visual features obtained from cross-view and cross-modal contrastive learning on images and point clouds. On five real-world datasets and on five tasks, we train and evaluate 108 models based on four pretraining variations. We find that cross-modal representation alignment discards complementary visual information, such as color and texture, and instead emphasizes redundant depth cues. The depth cues obtained from pretraining improve downstream depth prediction performance. Also overall, cross-modal alignment leads to more robust encoders than pre-training by cross-view alignment, especially on depth prediction, instance segmentation, and object detection.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2211.13309>)

[PDF](<https://arxiv.org/pdf/2211.13309.pdf>)
