# ObjectRelator: Enabling Cross-View Object Relation Understanding Across Ego-Centric and Exo-Centric Perspectives

Source: [ObjectRelator: Enabling Cross-View Object Relation Understanding Across Ego-Centric and Exo-Centric Perspectives](<https://www.sota2.com/research/paper/68fa7a361e209131cea81d0c>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T05:15:13.470Z

Authors: Yuqian Fu, Runze Wang, Bin Ren, Guolei Sun, Biao Gong, Yanwei Fu, Danda Pani Paudel, Xuanjing Huang, Luc Van Gool

Published: 2024-11-28T12:01:03.000Z

## Abstract

Bridging the gap between ego-centric and exo-centric views has been a long-standing question in computer vision. In this paper, we focus on the emerging Ego-Exo object correspondence task, which aims to understand object relations across ego-exo perspectives through segmentation. While numerous segmentation models have been proposed, most operate on a single image (view), making them impractical for cross-view scenarios. PSALM, a recently proposed segmentation method, stands out as a notable exception with its demonstrated zero-shot ability on this task. However, due to the drastic viewpoint change between ego and exo, PSALM fails to accurately locate and segment objects, especially in complex backgrounds or when object appearances change significantly. To address these issues, we propose ObjectRelator, a novel approach featuring two key modules: Multimodal Condition Fusion (MCFuse) and SSL-based Cross-View Object Alignment (XObjAlign). MCFuse introduces language as an additional cue, integrating both visual masks and textual descriptions to improve object localization and prevent incorrect associations. XObjAlign enforces cross-view consistency through self-supervised alignment, enhancing robustness to object appearance variations. Extensive experiments demonstrate ObjectRelator's effectiveness on the large-scale Ego-Exo4D benchmark and HANDAL-X (an adapted dataset for cross-view segmentation) with state-of-the-art performance. Code is made available at: http://yuqianfu.com/ObjectRelator.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Cross-view Object Segmentation on HANDAL-X](<https://www.sota2.com/research/sota/cross-view-object-segmentation-on-handal-x.md>): Cross-view Object Segmentation; HANDAL-X; IoU: 42.8.

- [Cross-view Instance Segmentation on Ego-Exo4D Ego-to-Exo](<https://www.sota2.com/research/sota/cross-view-instance-segmentation-on-ego-exo4d-ego-to-exo.md>): Cross-view Instance Segmentation; Ego-Exo4D Ego-to-Exo; IoU: 45.4.

- [Cross-view Instance Segmentation on Ego-Exo4D Exo-to-Ego](<https://www.sota2.com/research/sota/cross-view-instance-segmentation-on-ego-exo4d-exo-to-ego.md>): Cross-view Instance Segmentation; Ego-Exo4D Exo-to-Ego; IoU: 50.9.

- [Ego-to-Exo object correspondence on Ego-Exo4D Correspondences v2 (test)](<https://www.sota2.com/research/sota/ego-to-exo-object-correspondence-on-ego-exo4d-correspondences-v2-test.md>): Ego-to-Exo object correspondence; Ego-Exo4D Correspondences v2 (test); IoU: 35.3.

- [Exo-to-Ego object correspondence on Ego-Exo4D Correspondences v2 (test)](<https://www.sota2.com/research/sota/exo-to-ego-object-correspondence-on-ego-exo4d-correspondences-v2-test.md>): Exo-to-Ego object correspondence; Ego-Exo4D Correspondences v2 (test); IoU: 40.3.

- [Cross-view Object Correspondence on Ego-Exo4D v2 (test)](<https://www.sota2.com/research/sota/cross-view-object-correspondence-on-ego-exo4d-v2-test.md>): Cross-view Object Correspondence; Ego-Exo4D v2 (test); Ego Query IoU: not recorded.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2411.19083>)

[PDF](<https://arxiv.org/pdf/2411.19083.pdf>)
