# Your Embedding Model is SMARTer Than You Think

Source: [Your Embedding Model is SMARTer Than You Think](<https://www.sota2.com/research/paper/6a152916dbd7150506f7ded0>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-05-29T06:00:02.251Z

Authors: Jianrui Zhang, Hyun Jung Lee, Sukanta Ganguly, Tae-Eui Kam, Donghyun Kim, Yong Jae Lee

Published: 2026-05-24T08:36:34.000Z

## Abstract

Multimodal retrieval relies heavily on single-vector retrievers, which compress rich, sequential token sequences into one single global representation. While efficient, they discard fine-grained, local evidence critical for dense retrieval tasks. Multi-vector approaches were introduced as a solution, but they strictly require training and many ignore the necessity of a globally summarizing representation. To address this, we introduce SMART, a framework that unlocks the latent multi-vector capabilities of standard single-vector models. We first demonstrate that standard contrastive training on the pooled embedding implicitly shapes the retrieval geometry of preceding hidden states via gradient flow. By applying direct late-interaction over these frozen hidden states during inference, SMART acts as a plug-and-play upgrade that consistently improves performance across diverse modalities, improving even the state-of-the-art models further on MMEB-V2. We also reveal SMART's superior performance, as simple lightweight post-training not only saves time and compute, but also brings forth further improvement on Visual Document retrieval, allowing a single-vector model to outperform SoTA multi-vector counterparts. Ultimately, SMART offers both a highly efficient inference enhancement and a powerful finetuning technique for multimodal retrieval. We open source our code and weights at https://github.com/HanSolo9682/SMART.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Multimodal Dense Retrieval on MMEB V2](<https://www.sota2.com/research/sota/multimodal-dense-retrieval-on-mmeb-v2.md>): Multimodal Dense Retrieval; MMEB v2; Image Recall@1: 80.15.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2605.24938>)

[PDF](<https://arxiv.org/pdf/2605.24938.pdf>)

[Implementation](<https://github.com/HanSolo9682/SMART>)