# What-Meets-Where: Unified Learning of Action and Contact Localization in Images

Source: [What-Meets-Where: Unified Learning of Action and Contact Localization in Images](<https://www.sota2.com/research/paper/68fa7a8f1e209131cea83e1b>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-04-01T05:00:04.436Z

Authors: Yuxiao Wang, Yu Lei, Wolin Liang, Weiying Xue, Zhenao Wei, Nan Zhuang, Qi Liu

Published: 2025-08-13T02:06:33.000Z

## Abstract

People control their bodies to establish contact with the environment. To comprehensively understand actions across diverse visual contexts, it is essential to simultaneously consider \\textbf{what} action is occurring and \\textbf{where} it is happening. Current methodologies, however, often inadequately capture this duality, typically failing to jointly model both action semantics and their spatial contextualization within scenes. To bridge this gap, we introduce a novel vision task that simultaneously predicts high-level action semantics and fine-grained body-part contact regions. Our proposed framework, PaIR-Net, comprises three key components: the Contact Prior Aware Module (CPAM) for identifying contact-relevant body parts, the Prior-Guided Concat Segmenter (PGCS) for pixel-wise contact segmentation, and the Interaction Inference Module (IIM) responsible for integrating global interaction relationships. To facilitate this task, we present PaIR (Part-aware Interaction Representation), a comprehensive dataset containing 13,979 images that encompass 654 actions, 80 object categories, and 17 body parts. Experimental evaluation demonstrates that PaIR-Net significantly outperforms baseline approaches, while ablation studies confirm the efficacy of each architectural component. The code and dataset will be released upon publication.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Interaction and Contact Segmentation on PaIR-1](<https://www.sota2.com/research/sota/interaction-and-contact-segmentation-on-pair-1.md>): Interaction and Contact Segmentation; PaIR-1; mAP: 38.07.

- [Contact Region Segmentation on PaIR-2](<https://www.sota2.com/research/sota/contact-region-segmentation-on-pair-2.md>): Contact Region Segmentation; PaIR-2; SC Accuracy: 24.01.

- [Interaction Detection on PaIR-2](<https://www.sota2.com/research/sota/interaction-detection-on-pair-2.md>): Interaction Detection; PaIR-2; mAP: 61.32.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2508.09428>)

[PDF](<https://arxiv.org/pdf/2508.09428.pdf>)