# ReARTeR: Retrieval-Augmented Reasoning with Trustworthy Process Rewarding Source: [ReARTeR: Retrieval-Augmented Reasoning with Trustworthy Process Rewarding]() Collection index: [llms.txt]() Data updated: 2025-10-26T03:58:19.729Z Authors: Zhongxiang Sun, Qipeng Wang, Weijie Yu, Xiaoxue Zang, Kai Zheng, Jun Xu, Xiao Zhang, Song Yang, Han Li Published: 2025-01-14T05:56:26.000Z ## Abstract Retrieval-Augmented Generation (RAG) systems for Large Language Models (LLMs) hold promise in knowledge-intensive tasks but face limitations in complex multi-step reasoning. While recent methods have integrated RAG with chain-of-thought reasoning or test-time search using Process Reward Models (PRMs), these approaches encounter challenges such as a lack of explanations, bias in PRM training data, early-step bias in PRM scores, and insufficient post-training optimization of reasoning potential. To address these issues, we propose Retrieval-Augmented Reasoning through Trustworthy Process Rewarding (ReARTeR), a framework that enhances RAG systems' reasoning capabilities through post-training and test-time scaling. At test time, ReARTeR introduces Trustworthy Process Rewarding via a Process Reward Model for accurate scalar scoring and a Process Explanation Model (PEM) for generating natural language explanations, enabling step refinement. During post-training, it utilizes Monte Carlo Tree Search guided by Trustworthy Process Rewarding to collect high-quality step-level preference data, optimized through Iterative Preference Optimization. ReARTeR addresses three core challenges: (1) misalignment between PRM and PEM, tackled through off-policy preference learning; (2) bias in PRM training data, mitigated by balanced annotation methods and stronger annotations for challenging examples; and (3) early-step bias in PRM, resolved through a temporal-difference-based look-ahead search strategy. Experimental results on multi-step reasoning benchmarks demonstrate significant improvements, underscoring ReARTeR's potential to advance the reasoning capabilities of RAG systems. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Multi-hop Question Answering on Musique in-domain (test)](): Multi-hop Question Answering; Musique in-domain (test); Accuracy R: 29.6. - [Multi-hop Question Answering on 2WikiMultiHopQA in-domain (test)](): Multi-hop Question Answering; 2WikiMultiHopQA in-domain (test); Accuracy (Response): 55.4. - [Multi-hop Question Answering on Bamboogle out-of-domain (test)](): Multi-hop Question Answering; Bamboogle out-of-domain (test); Accuracy (R): 49.6. - [Multi-hop Question Answering on HotpotQA in-domain (test)](): Multi-hop Question Answering; HotpotQA in-domain (test); ACC\_R: 46.8. ## Original sources and implementation [Original paper]() [PDF]()