# ReST meets ReAct: Self-Improvement for Multi-Step Reasoning LLM Agent

Source: [ReST meets ReAct: Self-Improvement for Multi-Step Reasoning LLM Agent](<https://www.sota2.com/research/paper/68fa71771e209131cea516a5>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-01-04T03:58:00.967Z

Authors: Renat Aksitov, Sobhan Miryoosefi, Zonglin Li, Daliang Li, Sheila Babayan, Kavya Kopparapu, Zachary Fisher, Ruiqi Guo, Sushant Prakash, Pranesh Srinivasan, Manzil Zaheer, Felix Yu, Sanjiv Kumar

Published: 2023-12-15T18:20:15.000Z

## Abstract

Answering complex natural language questions often necessitates multi-step reasoning and integrating external information. Several systems have combined knowledge retrieval with a large language model (LLM) to answer such questions. These systems, however, suffer from various failure cases, and we cannot directly train them end-to-end to fix such failures, as interaction with external knowledge is non-differentiable. To address these deficiencies, we define a ReAct-style LLM agent with the ability to reason and act upon external knowledge. We further refine the agent through a ReST-like method that iteratively trains on previous trajectories, employing growing-batch reinforcement learning with AI feedback for continuous self-improvement and self-distillation. Starting from a prompted large model and after just two iterations of the algorithm, we can produce a fine-tuned small model that achieves comparable performance on challenging compositional question-answering benchmarks with two orders of magnitude fewer parameters.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Multi-step Reasoning on Bamboogle auto-eval (test)](<https://www.sota2.com/research/sota/multi-step-reasoning-on-bamboogle-auto-eval-test.md>): Multi-step Reasoning; Bamboogle auto-eval (test); Mean Accuracy: 76.1.

- [Multi-step reasoning on BamTwoogle (test)](<https://www.sota2.com/research/sota/multi-step-reasoning-on-bamtwoogle-test.md>): Multi-step Reasoning; BamTwoogle (test); Accuracy: 74.

- [Multi-step reasoning on Bamboogle (test)](<https://www.sota2.com/research/sota/multi-step-reasoning-on-bamboogle-test.md>): Multi-step Reasoning; Bamboogle (test); Accuracy: 74.4.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2312.10003>)

[PDF](<https://arxiv.org/pdf/2312.10003.pdf>)
