Text-based Person Search without Parallel Image-Text Data
About
Text-based person search (TBPS) aims to retrieve the images of the target person from a large image gallery based on a given natural language description. Existing methods are dominated by training models with parallel image-text pairs, which are very costly to collect. In this paper, we make the first attempt to explore TBPS without parallel image-text data ($\mu$-TBPS), in which only non-parallel images and texts, or even image-only data, can be adopted. Towards this end, we propose a two-stage framework, generation-then-retrieval (GTR), to first generate the corresponding pseudo text for each image and then perform the retrieval in a supervised manner. In the generation stage, we propose a fine-grained image captioning strategy to obtain an enriched description of the person image, which firstly utilizes a set of instruction prompts to activate the off-the-shelf pretrained vision-language model to capture and generate fine-grained person attributes, and then converts the extracted attributes into a textual description via the finetuned large language model or the hand-crafted template. In the retrieval stage, considering the noise interference of the generated texts for training model, we develop a confidence score-based training scheme by enabling more reliable texts to contribute more during the training. Experimental results on multiple TBPS benchmarks (i.e., CUHK-PEDES, ICFG-PEDES and RSTPReid) show that the proposed GTR can achieve a promising performance without relying on parallel image-text data.
Key insights
What problem does this TBPS paper address?
Text-based person search (TBPS) retrieves person images using text descriptions. The paper studies TBPS without costly parallel image-text pairs, proposing a generation-then-retrieval framework that first creates pseudo text descriptions from images, then trains a retrieval model with confidence-weighted losses.
How does this differ from prior TBPS work?
Unlike prior TBPS methods requiring parallel image-text pairs, this is the first to use only non-parallel or image-only data. On CUHK-PEDES, GTR achieves 48.49% R@1 without parallel data versus 73.38% by IRRA with parallel data, evaluated across three benchmarks.
What are the practical use cases?
The paper mentions potential applications in intelligent surveillance, safety protection, and smart city scenarios such as searching for suspects or lost individuals. No actual production deployment is documented; results are benchmark demonstrations on CUHK-PEDES, ICFG-PEDES, and RSTPReid.
What limitations do the authors acknowledge?
GTR shows a performance gap versus parallel-data methods, with weaker results on ICFG-PEDES because its longer texts exceed the limited attributes GTR's instruction prompts capture. The authors also note privacy concerns from surveillance images lacking formal consent.