# ChatSOP: An SOP-Guided MCTS Planning Framework for Controllable LLM Dialogue Agents

Source: [ChatSOP: An SOP-Guided MCTS Planning Framework for Controllable LLM Dialogue Agents](<https://www.sota2.com/research/paper/68fa771f1e209131cea7135b>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-06-05T05:00:04.417Z

Authors: Zhigen Li, Jianxiang Peng, Yanmeng Wang, Yong Cao, Tianhao Shen, Minghui Zhang, Linxi Su, Shang Wu, Yihang Wu, Yuqian Wang, Ye Wang, Wei Hu, Jianfeng Li, Shaojun Wang, Jing Xiao, Deyi Xiong

Published: 2024-07-04T12:23:02.000Z

## Abstract

Dialogue agents powered by Large Language Models (LLMs) show superior performance in various tasks. Despite the better user understanding and human-like responses, their \*\*lack of controllability\*\* remains a key challenge, often leading to unfocused conversations or task failure. To address this, we introduce Standard Operating Procedure (SOP) to regulate dialogue flow. Specifically, we propose \*\*ChatSOP\*\*, a novel SOP-guided Monte Carlo Tree Search (MCTS) planning framework designed to enhance the controllability of LLM-driven dialogue agents. To enable this, we curate a dataset comprising SOP-annotated multi-scenario dialogues, generated using a semi-automated role-playing system with GPT-4o and validated through strict manual quality control. Additionally, we propose a novel method that integrates Chain of Thought reasoning with supervised fine-tuning for SOP prediction and utilizes SOP-guided Monte Carlo Tree Search for optimal action planning during dialogues. Experimental results demonstrate the effectiveness of our method, such as achieving a 27.95% improvement in action accuracy compared to baseline models based on GPT-3.5 and also showing notable gains for open-source models. Dataset and codes are publicly available.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Dialogue Generation on Proposed Multi-scenario Dataset 1.0 (test)](<https://www.sota2.com/research/sota/dialogue-generation-on-proposed-multi-scenario-dataset-1-0-test.md>): Dialogue Generation; Proposed Multi-scenario Dataset 1.0 (test); Acc T: 86.37.

- [SOP Prediction on Proposed SOP dataset](<https://www.sota2.com/research/sota/sop-prediction-on-proposed-sop-dataset.md>): SOP Prediction; Proposed SOP dataset; Precision: 80.25.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2407.03884>)

[PDF](<https://arxiv.org/pdf/2407.03884.pdf>)