# Improving Interpretable Embeddings for Ad-hoc Video Search with Generative Captions and Multi-word Concept Bank

Source: [Improving Interpretable Embeddings for Ad-hoc Video Search with Generative Captions and Multi-word Concept Bank](<https://www.sota2.com/research/paper/68fa72cf1e209131cea58a82>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T02:34:49.549Z

Authors: Jiaxin Wu, Chong-Wah Ngo, Wing-Kwong Chan

Published: 2024-04-09T09:54:21.000Z

## Abstract

Aligning a user query and video clips in cross-modal latent space and that with semantic concepts are two mainstream approaches for ad-hoc video search (AVS). However, the effectiveness of existing approaches is bottlenecked by the small sizes of available video-text datasets and the low quality of concept banks, which results in the failures of unseen queries and the out-of-vocabulary problem. This paper addresses these two problems by constructing a new dataset and developing a multi-word concept bank. Specifically, capitalizing on a generative model, we construct a new dataset consisting of 7 million generated text and video pairs for pre-training. To tackle the out-of-vocabulary problem, we develop a multi-word concept bank based on syntax analysis to enhance the capability of a state-of-the-art interpretable AVS method in modeling relationships between query words. We also study the impact of current advanced features on the method. Experimental results show that the integration of the above-proposed elements doubles the R@1 performance of the AVS method on the MSRVTT dataset and improves the xinfAP on the TRECVid AVS query sets for 2016-2023 (eight years) by a margin from 2% to 77%, with an average about 20%.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Text-to-Video Retrieval on TRECVid IACC.3 2016](<https://www.sota2.com/research/sota/text-to-video-retrieval-on-trecvid-iacc-3-2016.md>): Text-to-Video Retrieval; TRECVid IACC.3 2016; xinfAP: 28.

- [Text-to-Video Retrieval on TRECVid IACC.3 2017 (tv17)](<https://www.sota2.com/research/sota/text-to-video-retrieval-on-trecvid-iacc-3-2017-tv17.md>): Text-to-Video Retrieval; TRECVid IACC.3 2017 (tv17); xinfAP: 34.9.

- [Text-to-Video Retrieval on TRECVid V3C1 2019 (tv19)](<https://www.sota2.com/research/sota/text-to-video-retrieval-on-trecvid-v3c1-2019-tv19.md>): Text-to-Video Retrieval; TRECVid V3C1 2019 (tv19); xinfAP: 24.2.

- [Text-to-Video Retrieval on TRECVid V3C1 2021 (tv21)](<https://www.sota2.com/research/sota/text-to-video-retrieval-on-trecvid-v3c1-2021-tv21.md>): Text-to-Video Retrieval; TRECVid V3C1 2021 (tv21); xinfAP: 36.5.

- [Text-to-Video Retrieval on TRECVid V3C2 2023 (tv23)](<https://www.sota2.com/research/sota/text-to-video-retrieval-on-trecvid-v3c2-2023-tv23.md>): Text-to-Video Retrieval; TRECVid V3C2 2023 (tv23); xinfAP: 29.5.

- [Text-to-Video Retrieval on TRECVid V3C1 2020 (tv20)](<https://www.sota2.com/research/sota/text-to-video-retrieval-on-trecvid-v3c1-2020-tv20.md>): Text-to-Video Retrieval; TRECVid V3C1 2020 (tv20); xinfAP: 0.352.

- [Text-to-Video Retrieval on TRECVid V3C2 2022](<https://www.sota2.com/research/sota/text-to-video-retrieval-on-trecvid-v3c2-2022.md>): Text-to-Video Retrieval; TRECVid V3C2 2022; xinfAP: 23.5.

- [Text-to-Video Retrieval on TRECVid IACC.3 2018 (tv18)](<https://www.sota2.com/research/sota/text-to-video-retrieval-on-trecvid-iacc-3-2018-tv18.md>): Text-to-Video Retrieval; TRECVid IACC.3 2018 (tv18); xinfAP: 16.5.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2404.06173>)

[PDF](<https://arxiv.org/pdf/2404.06173.pdf>)