# Modeling Temporal-Modal Entity Graph for Procedural Multimodal Machine Comprehension

Source: [Modeling Temporal-Modal Entity Graph for Procedural Multimodal Machine Comprehension](<https://www.sota2.com/research/paper/68fa6c2d1e209131cea3290a>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-01-04T05:15:06.401Z

Authors: Huibin Zhang, Zhengkun Zhang, Yao Zhang, Jun Wang, Yufan Li, Ning jiang, Xin wei, Zhenglu Yang

Published: 2022-04-06T03:41:13.000Z

## Abstract

Procedural Multimodal Documents (PMDs) organize textual instructions and corresponding images step by step. Comprehending PMDs and inducing their representations for the downstream reasoning tasks is designated as Procedural MultiModal Machine Comprehension (M3C). In this study, we approach Procedural M3C at a fine-grained level (compared with existing explorations at a document or sentence level), that is, entity. With delicate consideration, we model entity both in its temporal and cross-modal relation and propose a novel Temporal-Modal Entity Graph (TMEG). Specifically, graph structure is formulated to capture textual and visual entities and trace their temporal-modal evolution. In addition, a graph aggregation module is introduced to conduct graph encoding and reasoning. Comprehensive experiments across three Procedural M3C tasks are conducted on a traditional dataset RecipeQA and our new dataset CraftQA, which can better evaluate the generalization of TMEG.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Procedural Multimodal Machine Comprehension on RecipeQA (test)](<https://www.sota2.com/research/sota/procedural-multimodal-machine-comprehension-on-recipeqa-test.md>): Procedural Multimodal Machine Comprehension; RecipeQA (test); Cloze: 73.27.

- [Procedural Multimodal Machine Comprehension on CraftQA (test)](<https://www.sota2.com/research/sota/procedural-multimodal-machine-comprehension-on-craftqa-test.md>): Procedural Multimodal Machine Comprehension; CraftQA (test); Cloze: 48.01.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2204.02566>)

[PDF](<https://arxiv.org/pdf/2204.02566.pdf>)