# All in One Framework for Multimodal Re-identification in the Wild

Source: [All in One Framework for Multimodal Re-identification in the Wild](<https://www.sota2.com/research/paper/68fa732b1e209131cea5ae4d>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-01-04T04:51:25.620Z

Authors: He Li, Mang Ye, Ming Zhang, Bo Du

Published: 2024-05-08T01:04:36.000Z

## Abstract

In Re-identification (ReID), recent advancements yield noteworthy progress in both unimodal and cross-modal retrieval tasks. However, the challenge persists in developing a unified framework that could effectively handle varying multimodal data, including RGB, infrared, sketches, and textual information. Additionally, the emergence of large-scale models shows promising performance in various vision tasks but the foundation model in ReID is still blank. In response to these challenges, a novel multimodal learning paradigm for ReID is introduced, referred to as All-in-One (AIO), which harnesses a frozen pre-trained big model as an encoder, enabling effective multimodal retrieval without additional fine-tuning. The diverse multimodal data in AIO are seamlessly tokenized into a unified space, allowing the modality-shared frozen encoder to extract identity-consistent features comprehensively across all modalities. Furthermore, a meticulously crafted ensemble of cross-modality heads is designed to guide the learning trajectory. AIO is the \\textbf{first} framework to perform all-in-one ReID, encompassing four commonly used modalities. Experiments on cross-modal and multimodal ReID reveal that AIO not only adeptly handles various modal data but also excels in challenging contexts, showcasing exceptional performance in zero-shot and domain generalization scenarios.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Infrared-to-RGB Retrieval on Multimodal ReID I→R](<https://www.sota2.com/research/sota/infrared-to-rgb-retrieval-on-multimodal-reid-i-r.md>): Infrared-to-RGB Retrieval; Multimodal ReID I→R; Rank-1: 57.6.

- [Sketch-to-RGB Retrieval on Multimodal ReID S→R](<https://www.sota2.com/research/sota/sketch-to-rgb-retrieval-on-multimodal-reid-s-r.md>): Sketch-to-RGB Retrieval; Multimodal ReID S→R; Rank-1: 0.702.

- [RGB-to-RGB Retrieval on Multimodal ReID R→R](<https://www.sota2.com/research/sota/rgb-to-rgb-retrieval-on-multimodal-reid-r-r.md>): RGB-to-RGB Retrieval; Multimodal ReID R→R; Rank-1: 0.796.

- [Text-to-RGB Retrieval on Multimodal ReID T→R](<https://www.sota2.com/research/sota/text-to-rgb-retrieval-on-multimodal-reid-t-r.md>): Text-to-RGB Retrieval; Multimodal ReID T→R; Rank-1: 53.4.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2405.04741>)

[PDF](<https://arxiv.org/pdf/2405.04741.pdf>)

[Implementation](<https://github.com/lihe404/AIO>)
