# Relationship-based Neural Baby Talk

Source: [Relationship-based Neural Baby Talk](<https://www.sota2.com/research/paper/68fa69711e209131cea22152>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-25T23:00:47.140Z

Authors: Fan Fu, Tingting Xie, Ioannis Patras, Sepehr Jalali

Published: 2021-03-08T15:51:24.000Z

## Abstract

Understanding interactions between objects in an image is an important element for generating captions. In this paper, we propose a relationship-based neural baby talk (R-NBT) model to comprehensively investigate several types of pairwise object interactions by encoding each image via three different relationship-based graph attention networks (GATs). We study three main relationships: \\textit{spatial relationships} to explore geometric interactions, \\textit{semantic relationships} to extract semantic interactions, and \\textit{implicit relationships} to capture hidden information that could not be modelled explicitly as above. We construct three relationship graphs with the objects in an image as nodes, and the mutual relationships of pairwise objects as edges. By exploring features of neighbouring regions individually via GATs, we integrate different types of relationships into visual features of each node. Experiments on COCO dataset show that our proposed R-NBT model outperforms state-of-the-art models trained on COCO dataset in three image caption generation tasks.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2103.04846>)

[PDF](<https://arxiv.org/pdf/2103.04846.pdf>)