# Look, Listen and Segment: Towards Weakly Supervised Audio-visual Semantic Segmentation

Source: [Look, Listen and Segment: Towards Weakly Supervised Audio-visual Semantic Segmentation](<https://www.sota2.com/research/paper/69c21acf0636e5bf4afc4c84>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-03-25T05:00:19.379Z

Authors: Chengzhi Li, Heyan Huang, Ping Jian, Yanghao Zhou

Published: 2026-03-23T13:07:48.000Z

## Abstract

Audio-Visual Semantic Segmentation (AVSS) aligns audio and video at the pixel level but requires costly per-frame annotations. We introduce Weakly Supervised Audio-Visual Semantic Segmentation (WSAVSS), which uses only video-level labels to generate per-frame semantic masks of sounding objects. We decompose WSAVSS into looking, listening, and segmentation, and propose Progressive Cross-modal Alignment for Semantics (PCAS) with two modules: \*Looking-before-Listening\* and \*Listening-before-Segmentation\*. PCAS builds a classification task to train the audio-visual encoder using video labels, injects visual semantic prompts to enhance frame-level audio understanding, and then applies progressive contrastive alignment to map audio categories to image regions without mask annotations. Experiments show PCAS achieves state-of-the-art performance among weakly supervised methods on AVS and remains competitive with fully supervised baselines on AVSS, validating its effectiveness.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2603.21948>)

[PDF](<https://arxiv.org/pdf/2603.21948.pdf>)