Weakly-supervised Automated Audio Captioning via text only training
About
In recent years, datasets of paired audio and captions have enabled remarkable success in automatically generating descriptions for audio clips, namely Automated Audio Captioning (AAC). However, it is labor-intensive and time-consuming to collect a sufficient number of paired audio and captions. Motivated by the recent advances in Contrastive Language-Audio Pretraining (CLAP), we propose a weakly-supervised approach to train an AAC model assuming only text data and a pre-trained CLAP model, alleviating the need for paired target data. Our approach leverages the similarity between audio and text embeddings in CLAP. During training, we learn to reconstruct the text from the CLAP text embedding, and during inference, we decode using the audio embeddings. To mitigate the modality gap between the audio and text embeddings we employ strategies to bridge the gap during training and inference stages. We evaluate our proposed method on Clotho and AudioCaps datasets demonstrating its ability to achieve a relative performance of up to ~$83\%$ compared to fully supervised approaches trained with paired target data.
Key insights
What problem does this weakly-supervised captioning approach solve?
Automated Audio Captioning normally needs paired audio-text data, which is scarce and costly. The authors train a decoder on text-only captions plus a frozen CLAP model, decoding audio embeddings at inference to generate captions without paired training data.
How does this differ from prior fully supervised methods?
No prior work addressed AAC in similar weakly-supervised settings, so the authors compare against fully supervised models. Their best method, projection-based decoding, reaches 80% and 83% of fully supervised SPIDEr on Clotho and AudioCaps respectively.
What practical use does this text-only training enable?
It enables domain adaptation for audio captioning using only target-domain text, removing the need to collect paired audio-caption data. The paper demonstrates this on Clotho and AudioCaps benchmarks but documents no production deployment.
What limitations and future work do the authors state?
The modality gap between audio and text embeddings makes direct decoding suboptimal, requiring mitigation strategies. The authors plan to extend the approach to music captioning and audio question answering, and to train a mapping network to learn the gap in a supervised manner.