# Estimated Audio-Caption Correspondences Improve Language-Based Audio Retrieval

Source: [Estimated Audio-Caption Correspondences Improve Language-Based Audio Retrieval](<https://www.sota2.com/research/paper/68fa748f1e209131cea627b6>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-01-04T04:03:22.633Z

Authors: Paul Primus, Florian Schmid, Gerhard Widmer

Published: 2024-08-21T14:10:58.000Z

## Abstract

Dual-encoder-based audio retrieval systems are commonly optimized with contrastive learning on a set of matching and mismatching audio-caption pairs. This leads to a shared embedding space in which corresponding items from the two modalities end up close together. Since audio-caption datasets typically only contain matching pairs of recordings and descriptions, it has become common practice to create mismatching pairs by pairing the audio with a caption randomly drawn from the dataset. This is not ideal because the randomly sampled caption could, just by chance, partly or entirely describe the audio recording. However, correspondence information for all possible pairs is costly to annotate and thus typically unavailable; we, therefore, suggest substituting it with estimated correspondences. To this end, we propose a two-staged training procedure in which multiple retrieval models are first trained as usual, i.e., without estimated correspondences. In the second stage, the audio-caption correspondences predicted by these models then serve as prediction targets. We evaluate our method on the ClothoV2 and the AudioCaps benchmark and show that it improves retrieval performance, even in a restricting self-distillation setting where a single model generates and then learns from the estimated correspondences. We further show that our method outperforms the current state of the art by 1.6 pp. mAP@10 on the ClothoV2 benchmark.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Audio Retrieval on Clotho V2 (val)](<https://www.sota2.com/research/sota/audio-retrieval-on-clotho-v2-val.md>): Audio Retrieval; Clotho V2 (val); R@1: 27.69.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2408.11641>)

[PDF](<https://arxiv.org/pdf/2408.11641.pdf>)

[Implementation](<https://github.com/OptimusPrimus/salsa>)