# Learning to Separate Object Sounds by Watching Unlabeled Video

Source: [Learning to Separate Object Sounds by Watching Unlabeled Video](<https://www.sota2.com/research/paper/68fa649b1e209131cea0584a>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-25T21:08:02.971Z

Authors: Ruohan Gao, Rogerio Feris, Kristen Grauman

Published: 2018-04-05T04:06:46.000Z

## Abstract

Perceiving a scene most fully requires all the senses. Yet modeling how objects look and sound is challenging: most natural scenes and events contain multiple objects, and the audio track mixes all the sound sources together. We propose to learn audio-visual object models from unlabeled video, then exploit the visual context to perform audio source separation in novel videos. Our approach relies on a deep multi-instance multi-label learning framework to disentangle the audio frequency bases that map to individual visual objects, even without observing/hearing those objects in isolation. We show how the recovered disentangled bases can be used to guide audio source separation to obtain better-separated, object-level sounds. Our work is the first to learn audio source separation from large-scale "in the wild" videos containing multiple audio sources per video. We obtain state-of-the-art results on visually-aided audio source separation and audio denoising. Our video results: http://vision.cs.utexas.edu/projects/separating\_object\_sounds/

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Audio Source Separation on AudioSet SingleSource (test)](<https://www.sota2.com/research/sota/audio-source-separation-on-audioset-singlesource-test.md>): Audio Source Separation; AudioSet SingleSource (test); SDR: 1.83.

- [Visually-assisted audio denoising on AV-Bench Wooden Horse](<https://www.sota2.com/research/sota/visually-assisted-audio-denoising-on-av-bench-wooden-horse.md>): Visually-assisted audio denoising; AV-Bench Wooden Horse; NSDR (dB): 12.3.

- [Visually-assisted audio denoising on AV-Bench Violin Yanni](<https://www.sota2.com/research/sota/visually-assisted-audio-denoising-on-av-bench-violin-yanni.md>): Visually-assisted audio denoising; AV-Bench Violin Yanni; NSDR (dB): 7.88.

- [Visually-assisted audio denoising on AV-Bench Guitar Solo](<https://www.sota2.com/research/sota/visually-assisted-audio-denoising-on-av-bench-guitar-solo.md>): Visually-assisted audio denoising; AV-Bench Guitar Solo; NSDR (dB): 11.4.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/1804.01665>)

[PDF](<https://arxiv.org/pdf/1804.01665.pdf>)
