# Multi-source Semantic Graph-based Multimodal Sarcasm Explanation Generation

Source: [Multi-source Semantic Graph-based Multimodal Sarcasm Explanation Generation](<https://www.sota2.com/research/paper/68fa6fea1e209131cea48245>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-01-02T03:05:48.287Z

Authors: Liqiang Jing, Xuemeng Song, Kun Ouyang, Mengzhao Jia, Liqiang Nie

Published: 2023-06-29T03:26:10.000Z

## Abstract

Multimodal Sarcasm Explanation (MuSE) is a new yet challenging task, which aims to generate a natural language sentence for a multimodal social post (an image as well as its caption) to explain why it contains sarcasm. Although the existing pioneer study has achieved great success with the BART backbone, it overlooks the gap between the visual feature space and the decoder semantic space, the object-level metadata of the image, as well as the potential external knowledge. To solve these limitations, in this work, we propose a novel mulTi-source sEmantic grAph-based Multimodal sarcasm explanation scheme, named TEAM. In particular, TEAM extracts the object-level semantic meta-data instead of the traditional global visual features from the input image. Meanwhile, TEAM resorts to ConceptNet to obtain the external related knowledge concepts for the input text and the extracted object meta-data. Thereafter, TEAM introduces a multi-source semantic graph that comprehensively characterize the multi-source (i.e., caption, object meta-data, external knowledge) semantic relations to facilitate the sarcasm reasoning. Extensive experiments on a public released dataset MORE verify the superiority of our model over cutting-edge methods.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Sarcasm Explanation Generation on MORE Non-OCR samples 1.0](<https://www.sota2.com/research/sota/sarcasm-explanation-generation-on-more-non-ocr-samples-1-0.md>): Sarcasm explanation generation; MORE Non-OCR samples 1.0; BLEU-1: 56.45.

- [Sarcasm Explanation Generation on MORE OCR samples (test)](<https://www.sota2.com/research/sota/sarcasm-explanation-generation-on-more-ocr-samples-test.md>): Sarcasm explanation generation; MORE OCR samples (test); ROUGE-1 Precision: 52.88.

- [Sarcasm Explanation Generation on MORE 1.0 (All samples)](<https://www.sota2.com/research/sota/sarcasm-explanation-generation-on-more-1-0-all-samples.md>): Sarcasm explanation generation; MORE 1.0 (All samples); BLEU-1: 55.32.

- [Sarcasm Explanation Generation on MORE OCR samples 1.0](<https://www.sota2.com/research/sota/sarcasm-explanation-generation-on-more-ocr-samples-1-0.md>): Sarcasm explanation generation; MORE OCR samples 1.0; ROUGE-1 (B1): 52.88.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2306.16650>)

[PDF](<https://arxiv.org/pdf/2306.16650.pdf>)

[Implementation](<https://github.com/LiqiangJing/TEAM>)
