# Learning Word-Like Units from Joint Audio-Visual Analysis

Source: [Learning Word-Like Units from Joint Audio-Visual Analysis](<https://www.sota2.com/research/paper/68fa63681e209131ce9fe7db>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-25T20:40:16.176Z

Authors: David Harwath, James R. Glass

Published: 2017-01-25T20:40:56.000Z

## Abstract

Given a collection of images and spoken audio captions, we present a method for discovering word-like acoustic units in the continuous speech signal and grounding them to semantically relevant image regions. For example, our model is able to detect spoken instances of the word 'lighthouse' within an utterance and associate them with image regions containing lighthouses. We do not use any form of conventional automatic speech recognition, nor do we use any text transcriptions or conventional linguistic annotations. Our model effectively implements a form of spoken language acquisition, in which the computer learns not only to recognize word categories by sound, but also to enrich the words it learns with semantics by grounding them in images.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Image-to-Speech Retrieval on Places Audio Caption Dataset 1,000 image/caption (held out)](<https://www.sota2.com/research/sota/image-to-speech-retrieval-on-places-audio-caption-dataset-1-000-image-caption-held-out.md>): Image-to-Speech Retrieval; Places audio caption dataset 1,000 image/caption (held out); R@1: 13.

- [Image-to-Audio Retrieval on PlacesAudio (val)](<https://www.sota2.com/research/sota/image-to-audio-retrieval-on-placesaudio-val.md>): Image-to-Audio Retrieval; PlacesAudio (val); Acc @10: 54.2.

- [Audio-to-Image Retrieval on PlacesAudio (val)](<https://www.sota2.com/research/sota/audio-to-image-retrieval-on-placesaudio-val.md>): Audio-to-image retrieval; PlacesAudio (val); Acc @10: 56.4.

- [Speech-to-Image Retrieval on Places audio caption dataset 1,000 image/caption (held out)](<https://www.sota2.com/research/sota/speech-to-image-retrieval-on-places-audio-caption-dataset-1-000-image-caption-held-out.md>): Speech-to-Image Retrieval; Places audio caption dataset 1,000 image/caption (held out); R@1: 0.161.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/1701.07481>)

[PDF](<https://arxiv.org/pdf/1701.07481.pdf>)