# Procedural Reasoning Networks for Understanding Multimodal Procedures

Source: [Procedural Reasoning Networks for Understanding Multimodal Procedures](<https://www.sota2.com/research/paper/68fa66661e209131cea101b3>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-27T05:59:47.071Z

Authors: Mustafa Sercan Amac, Semih Yagcioglu, Aykut Erdem, Erkut Erdem

Published: 2019-09-19T08:39:00.000Z

## Abstract

This paper addresses the problem of comprehending procedural commonsense knowledge. This is a challenging task as it requires identifying key entities, keeping track of their state changes, and understanding temporal and causal relations. Contrary to most of the previous work, in this study, we do not rely on strong inductive bias and explore the question of how multimodality can be exploited to provide a complementary semantic signal. Towards this end, we introduce a new entity-aware neural comprehension model augmented with external relational memory units. Our model learns to dynamically update entity states in relation to each other while reading the text instructions. Our experimental analysis on the visual reasoning tasks in the recently proposed RecipeQA dataset reveals that our approach improves the accuracy of the previously reported models by a large margin. Moreover, we find that our model learns effective dynamic representations of entities even though we do not use any supervision at the level of entity states.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Procedural Multimodal Machine Comprehension on RecipeQA (test)](<https://www.sota2.com/research/sota/procedural-multimodal-machine-comprehension-on-recipeqa-test.md>): Procedural Multimodal Machine Comprehension; RecipeQA (test); Cloze: 77.6.

- [Procedural Multimodal Machine Comprehension on CraftQA (test)](<https://www.sota2.com/research/sota/procedural-multimodal-machine-comprehension-on-craftqa-test.md>): Procedural Multimodal Machine Comprehension; CraftQA (test); Cloze: 55.33.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/1909.08859>)

[PDF](<https://arxiv.org/pdf/1909.08859.pdf>)