# Image Captioning for Effective Use of Language Models in Knowledge-Based Visual Question Answering

Source: [Image Captioning for Effective Use of Language Models in Knowledge-Based Visual Question Answering](<https://www.sota2.com/research/paper/68fa6d651e209131cea39754>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T00:32:23.745Z

Authors: Ander Salaberria, Gorka Azkune, Oier Lopez de Lacalle, Aitor Soroa, Eneko Agirre

Published: 2021-09-15T14:11:29.000Z

## Abstract

Integrating outside knowledge for reasoning in visio-linguistic tasks such as visual question answering (VQA) is an open problem. Given that pretrained language models have been shown to include world knowledge, we propose to use a unimodal (text-only) train and inference procedure based on automatic off-the-shelf captioning of images and pretrained language models. Our results on a visual question answering task which requires external knowledge (OK-VQA) show that our text-only model outperforms pretrained multimodal (image-text) models of comparable number of parameters. In contrast, our model is less effective in a standard VQA task (VQA 2.0) confirming that our text-only method is specially effective for tasks requiring external knowledge. In addition, we show that increasing the language model's size improves notably its performance, yielding results comparable to the state-of-the-art with our largest model, significantly outperforming current multimodal systems, even though augmented with external knowledge. Our qualitative analysis on OK-VQA reveals that automatic captions often fail to capture relevant information in the images, which seems to be balanced by the better inference ability of the text-only language models. Our work opens up possibilities to further improve inference in visio-linguistic tasks

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Visual Question Answering on OK-VQA (test)](<https://www.sota2.com/research/sota/visual-question-answering-on-ok-vqa-test.md>): Visual Question Answering; OK-VQA (test); Accuracy: 39.2.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2109.08029>)

[PDF](<https://arxiv.org/pdf/2109.08029.pdf>)