# Improved Instruction Ordering in Recipe-Grounded Conversation

Source: [Improved Instruction Ordering in Recipe-Grounded Conversation](<https://www.sota2.com/research/paper/68fa6f781e209131cea45c7c>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-01-02T02:41:15.017Z

Authors: Duong Minh Le, Ruohao Guo, Wei Xu, Alan Ritter

Published: 2023-05-26T21:57:11.000Z

## Abstract

In this paper, we study the task of instructional dialogue and focus on the cooking domain. Analyzing the generated output of the GPT-J model, we reveal that the primary challenge for a recipe-grounded dialog system is how to provide the instructions in the correct order. We hypothesize that this is due to the model's lack of understanding of user intent and inability to track the instruction state (i.e., which step was last instructed). Therefore, we propose to explore two auxiliary subtasks, namely User Intent Detection and Instruction State Tracking, to support Response Generation with improved instruction grounding. Experimenting with our newly collected dataset, ChattyChef, shows that incorporating user intent and instruction state information helps the response generation model mitigate the incorrect order issue. Furthermore, to investigate whether ChatGPT has completely solved this task, we analyze its outputs and find that it also makes mistakes (10.7% of the responses), about half of which are out-of-order instructions. We will release ChattyChef to facilitate further research in this area at: https://github.com/octaviaguo/ChattyChef.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [User Intent Detection on ChattyChef (test)](<https://www.sota2.com/research/sota/user-intent-detection-on-chattychef-test.md>): User Intent Detection; ChattyChef (test); Micro F1: 78.3.

- [Instruction State Tracking on ChattyChef (val)](<https://www.sota2.com/research/sota/instruction-state-tracking-on-chattychef-val.md>): Instruction State Tracking; ChattyChef (val); WordMatch: 0.82.

- [Instruction State Tracking on ChattyChef (test)](<https://www.sota2.com/research/sota/instruction-state-tracking-on-chattychef-test.md>): Instruction State Tracking; ChattyChef (test); WordMatch: 79.

- [Response Generation on ChattyChef (test)](<https://www.sota2.com/research/sota/response-generation-on-chattychef-test.md>): Response Generation; ChattyChef (test); BLEU: 4.7.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2305.17280>)

[PDF](<https://arxiv.org/pdf/2305.17280.pdf>)

[Implementation](<https://github.com/octaviaguo/ChattyChef>)