# Graph-Structured Representations for Visual Question Answering

Source: [Graph-Structured Representations for Visual Question Answering](<https://www.sota2.com/research/paper/68fa633a1e209131ce9fda20>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-01-04T02:50:26.211Z

Authors: Damien Teney, Lingqiao Liu, Anton van den Hengel

Published: 2016-09-19T05:21:36.000Z

## Abstract

This paper proposes to improve visual question answering (VQA) with structured representations of both scene contents and questions. A key challenge in VQA is to require joint reasoning over the visual and text domains. The predominant CNN/LSTM-based approach to VQA is limited by monolithic vector representations that largely ignore structure in the scene and in the form of the question. CNN feature vectors cannot effectively capture situations as simple as multiple object instances, and LSTMs process questions as series of words, which does not reflect the true complexity of language structure. We instead propose to build graphs over the scene objects and over the question words, and we describe a deep neural network that exploits the structure in these representations. This shows significant benefit over the sequential processing of LSTMs. The overall efficacy of our approach is demonstrated by significant improvements over the state-of-the-art, from 71.2% to 74.4% in accuracy on the "abstract scenes" multiple-choice benchmark, and from 34.7% to 39.1% in accuracy over pairs of "balanced" scenes, i.e. images with fine-grained differences and opposite yes/no answers to a same question.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Visual Question Answering on abstract scenes (test)](<https://www.sota2.com/research/sota/visual-question-answering-on-abstract-scenes-test.md>): Visual Question Answering; abstract scenes (test); Overall Accuracy (MC): 74.37.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/1609.05600>)

[PDF](<https://arxiv.org/pdf/1609.05600.pdf>)