# A Unified Audio-Visual Learning Framework for Localization, Separation, and Recognition

Source: [A Unified Audio-Visual Learning Framework for Localization, Separation, and Recognition](<https://www.sota2.com/research/paper/68fa6f981e209131cea4682a>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T01:23:24.960Z

Authors: Shentong Mo, Pedro Morgado

Published: 2023-05-30T23:53:12.000Z

## Abstract

The ability to accurately recognize, localize and separate sound sources is fundamental to any audio-visual perception task. Historically, these abilities were tackled separately, with several methods developed independently for each task. However, given the interconnected nature of source localization, separation, and recognition, independent models are likely to yield suboptimal performance as they fail to capture the interdependence between these tasks. To address this problem, we propose a unified audio-visual learning framework (dubbed OneAVM) that integrates audio and visual cues for joint localization, separation, and recognition. OneAVM comprises a shared audio-visual encoder and task-specific decoders trained with three objectives. The first objective aligns audio and visual representations through a localized audio-visual correspondence loss. The second tackles visual source separation using a traditional mix-and-separate framework. Finally, the third objective reinforces visual feature separation and localization by mixing images in pixel space and aligning their representations with those of all corresponding sound sources. Extensive experiments on MUSIC, VGG-Instruments, VGG-Music, and VGGSound datasets demonstrate the effectiveness of OneAVM for all three tasks, audio-visual source localization, separation, and nearest neighbor recognition, and empirically demonstrate a strong positive transfer between them.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Sound Source Separation on MUSIC](<https://www.sota2.com/research/sota/sound-source-separation-on-music.md>): Sound source separation; music; SDR: 7.38.

- [Sound source separation on VGGS-Instruments](<https://www.sota2.com/research/sota/sound-source-separation-on-vggs-instruments.md>): Sound source separation; VGGS-Instruments; SDR: 5.36.

- [Sound Source Separation on VGGS-Music](<https://www.sota2.com/research/sota/sound-source-separation-on-vggs-music.md>): Sound source separation; VGGS-Music; SDR: 2.51.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2305.19458>)

[PDF](<https://arxiv.org/pdf/2305.19458.pdf>)
