# TCIC: Theme Concepts Learning Cross Language and Vision for Image Captioning

Source: [TCIC: Theme Concepts Learning Cross Language and Vision for Image Captioning](<https://www.sota2.com/research/paper/68fa6a3e1e209131cea26ee7>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-25T23:20:37.043Z

Authors: Zhihao Fan, Zhongyu Wei, Siyuan Wang, Ruize Wang, Zejun Li, Haijun Shan, Xuanjing Huang

Published: 2021-06-21T09:12:55.000Z

## Abstract

Existing research for image captioning usually represents an image using a scene graph with low-level facts (objects and relations) and fails to capture the high-level semantics. In this paper, we propose a Theme Concepts extended Image Captioning (TCIC) framework that incorporates theme concepts to represent high-level cross-modality semantics. In practice, we model theme concepts as memory vectors and propose Transformer with Theme Nodes (TTN) to incorporate those vectors for image captioning. Considering that theme concepts can be learned from both images and captions, we propose two settings for their representations learning based on TTN. On the vision side, TTN is configured to take both scene graph based features and theme concepts as input for visual representation learning. On the language side, TTN is configured to take both captions and theme concepts as input for text representation re-construction. Both settings aim to generate target captions with the same transformer-based decoder. During the training, we further align representations of theme concepts learned from images and corresponding captions to enforce the cross-modality learning. Experimental results on MS COCO show the effectiveness of our approach compared to some state-of-the-art models.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Image Captioning on MS COCO Karpathy (test)](<https://www.sota2.com/research/sota/image-captioning-on-ms-coco-karpathy-test.md>): Image Captioning; MS COCO Karpathy (test); CIDEr: 1.354.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2106.10936>)

[PDF](<https://arxiv.org/pdf/2106.10936.pdf>)