# Passage Retrieval for Outside-Knowledge Visual Question Answering Source: [Passage Retrieval for Outside-Knowledge Visual Question Answering]() Collection index: [llms.txt]() Data updated: 2025-10-25T23:10:38.141Z Authors: Chen Qu, Hamed Zamani, Liu Yang, W. Bruce Croft, Erik Learned-Miller Published: 2021-05-09T13:27:22.000Z ## Abstract In this work, we address multi-modal information needs that contain text questions and images by focusing on passage retrieval for outside-knowledge visual question answering. This task requires access to outside knowledge, which in our case we define to be a large unstructured passage collection. We first conduct sparse retrieval with BM25 and study expanding the question with object names and image captions. We verify that visual clues play an important role and captions tend to be more informative than object names in sparse retrieval. We then construct a dual-encoder dense retriever, with the query encoder being LXMERT, a multi-modal pre-trained transformer. We further show that dense retrieval significantly outperforms sparse retrieval that uses object expansion. Moreover, dense retrieval matches the performance of sparse retrieval that leverages human-generated captions. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. ## Original sources and implementation [Original paper]() [PDF]()