# Process Reward Models for LLM Agents: Practical Framework and Directions Source: [Process Reward Models for LLM Agents: Practical Framework and Directions]() Collection index: [llms.txt]() Data updated: 2025-10-26T04:08:06.137Z Authors: Sanjiban Choudhury Published: 2025-02-14T17:34:28.000Z ## Abstract We introduce Agent Process Reward Models (AgentPRM), a simple and scalable framework for training LLM agents to continually improve through interactions. AgentPRM follows a lightweight actor-critic paradigm, using Monte Carlo rollouts to compute reward targets and optimize policies. It requires minimal modifications to existing RLHF pipelines, making it easy to integrate at scale. Beyond AgentPRM, we propose InversePRM, which learns process rewards directly from demonstrations without explicit outcome supervision. We also explore key challenges and opportunities, including exploration, process reward shaping, and model-predictive reasoning. We evaluate on ALFWorld benchmark, show that small 3B models trained with AgentPRM and InversePRM outperform strong GPT-4o baselines, and analyze test-time scaling, reward hacking, and more. Our code is available at: https://github.com/sanjibanc/agent\_prm. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Deep Search Reasoning on XBench DeepSearch2505](): Deep Search Reasoning; XBench DeepSearch2505; Score: 25. - [General AI Assistant Reasoning on GAIA-Text-103 1.0 (test)](): General AI Assistant Reasoning; GAIA-Text-103 1.0 (test); Overall Accuracy: 38.9. - [Interactive Decision-making on ALFWorld](): Interactive Decision-making; AlfWorld; Overall Success Rate: 73.9. ## Original sources and implementation [Original paper]() [PDF]()