# Speech-Image Semantic Alignment Does Not Depend on Any Prior Classification Tasks

Source: [Speech-Image Semantic Alignment Does Not Depend on Any Prior Classification Tasks](<https://www.sota2.com/research/paper/68fa577a1e209131ce9a5db4>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-25T14:45:29.807Z

Authors: Masood S. Mortazavi

Published: 2020-10-29T00:14:33.000Z

## Abstract

Semantically-aligned $(speech, image)$ datasets can be used to explore "visually-grounded speech". In a majority of existing investigations, features of an image signal are extracted using neural networks "pre-trained" on other tasks (e.g., classification on ImageNet). In still others, pre-trained networks are used to extract audio features prior to semantic embedding. Without "transfer learning" through pre-trained initialization or pre-trained feature extraction, previous results have tended to show low rates of recall in $speech \\rightarrow image$ and $image \\rightarrow speech$ queries.   Choosing appropriate neural architectures for encoders in the speech and image branches and using large datasets, one can obtain competitive recall rates without any reliance on any pre-trained initialization or feature extraction: $(speech,image)$ semantic alignment and $speech \\rightarrow image$ and $image \\rightarrow speech$ retrieval are canonical tasks worthy of independent investigation of their own and allow one to explore other questions---e.g., the size of the audio embedder can be reduced significantly with little loss of recall rates in $speech \\rightarrow image$ and $image \\rightarrow speech$ queries.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2010.15288>)

[PDF](<https://arxiv.org/pdf/2010.15288.pdf>)
