# Retrieve, Caption, Generate: Visual Grounding for Enhancing Commonsense in Text Generation Models

Source: [Retrieve, Caption, Generate: Visual Grounding for Enhancing Commonsense in Text Generation Models](<https://www.sota2.com/research/paper/68fa6c181e209131cea321ba>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T00:03:49.010Z

Authors: Steven Y. Feng, Kevin Lu, Zhuofu Tao, Malihe Alikhani, Teruko Mitamura, Eduard Hovy, Varun Gangal

Published: 2021-09-08T19:38:11.000Z

## Abstract

We investigate the use of multimodal information contained in images as an effective method for enhancing the commonsense of Transformer models for text generation. We perform experiments using BART and T5 on concept-to-text generation, specifically the task of generative commonsense reasoning, or CommonGen. We call our approach VisCTG: Visually Grounded Concept-to-Text Generation. VisCTG involves captioning images representing appropriate everyday scenarios, and using these captions to enrich and steer the generation process. Comprehensive evaluation and analysis demonstrate that VisCTG noticeably improves model performance while successfully addressing several issues of the baseline generations, including poor commonsense, fluency, and specificity.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Generative Commonsense Reasoning on CommonGen V1.0 (test)](<https://www.sota2.com/research/sota/generative-commonsense-reasoning-on-commongen-v1-0-test.md>): Generative Commonsense Reasoning; CommonGen V1.0 (test); BLEU-4: 27.67.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2109.03892>)

[PDF](<https://arxiv.org/pdf/2109.03892.pdf>)
