# Distilled Dual-Encoder Model for Vision-Language Understanding Source: [Distilled Dual-Encoder Model for Vision-Language Understanding]() Collection index: [llms.txt]() Data updated: 2025-10-26T00:40:22.216Z Authors: Zekun Wang, Wenhui Wang, Haichao Zhu, Ming Liu, Bing Qin, Furu Wei Published: 2021-12-16T09:21:18.000Z ## Abstract We propose a cross-modal attention distillation framework to train a dual-encoder model for vision-language understanding tasks, such as visual reasoning and visual question answering. Dual-encoder models have a faster inference speed than fusion-encoder models and enable the pre-computation of images and text during inference. However, the shallow interaction module used in dual-encoder models is insufficient to handle complex vision-language understanding tasks. In order to learn deep interactions of images and text, we introduce cross-modal attention distillation, which uses the image-to-text and text-to-image attention distributions of a fusion-encoder model to guide the training of our dual-encoder model. In addition, we show that applying the cross-modal attention distillation for both pre-training and fine-tuning stages achieves further improvements. Experimental results demonstrate that the distilled dual-encoder model achieves competitive performance for visual reasoning, visual entailment and visual question answering tasks while enjoying a much faster inference speed than fusion-encoder models. Our code and models will be publicly available at https://github.com/kugwzk/Distilled-DualEncoder. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Natural Language Visual Reasoning on NLVR2 (val)](): Natural Language Visual Reasoning; NLVR2 (val); Accuracy: 74.16. - [Visual Reasoning on NLVR2 (test)](): Visual Reasoning; NLVR2 (test); Accuracy: 74.3. - [Visual Question Answering on VQA (test-std)](): Visual Question Answering; VQA (test-std); Accuracy: 69.2. - [Visual Entailment on SNLI-VE (test)](): Visual Entailment; SNLI-VE (test); Overall Accuracy: 76.3. - [Natural Language Visual Reasoning on NLVR2 (test-p)](): Natural Language Visual Reasoning; NLVR2 (test-p); Accuracy: 75.6. - [Visual Question Answering on VQA 2.0 (test-dev)](): Visual Question Answering; VQA 2.0 (test-dev); Accuracy: 68.05. ## Original sources and implementation [Original paper]() [PDF]()