# SOrT-ing VQA Models : Contrastive Gradient Learning for Improved Consistency

Source: [SOrT-ing VQA Models : Contrastive Gradient Learning for Improved Consistency](<https://www.sota2.com/research/paper/68fa68d31e209131cea1e63d>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-25T22:46:52.640Z

Authors: Sameer Dharur, Purva Tendulkar, Dhruv Batra, Devi Parikh, Ramprasaath R. Selvaraju

Published: 2020-10-20T05:15:48.000Z

## Abstract

Recent research in Visual Question Answering (VQA) has revealed state-of-the-art models to be inconsistent in their understanding of the world -- they answer seemingly difficult questions requiring reasoning correctly but get simpler associated sub-questions wrong. These sub-questions pertain to lower level visual concepts in the image that models ideally should understand to be able to answer the higher level question correctly. To address this, we first present a gradient-based interpretability approach to determine the questions most strongly correlated with the reasoning question on an image, and use this to evaluate VQA models on their ability to identify the relevant sub-questions needed to answer a reasoning question. Next, we propose a contrastive gradient learning based approach called Sub-question Oriented Tuning (SOrT) which encourages models to rank relevant sub-questions higher than irrelevant questions for an \<image, reasoning-question\> pair. We show that SOrT improves model consistency by upto 6.5% points over existing baselines, while also improving visual grounding.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2010.10038>)

[PDF](<https://arxiv.org/pdf/2010.10038.pdf>)
