# Policy Gradient in Robust MDPs with Global Convergence Guarantee

Source: [Policy Gradient in Robust MDPs with Global Convergence Guarantee](<https://www.sota2.com/research/paper/68fa73331e209131cea5b184>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T02:44:18.752Z

Authors: Qiuhao Wang, Chin Pang Ho, Marek Petrik

Published: 2022-12-20T17:14:14.000Z

## Abstract

Robust Markov decision processes (RMDPs) provide a promising framework for computing reliable policies in the face of model errors. Many successful reinforcement learning algorithms build on variations of policy-gradient methods, but adapting these methods to RMDPs has been challenging. As a result, the applicability of RMDPs to large, practical domains remains limited. This paper proposes a new Double-Loop Robust Policy Gradient (DRPG), the first generic policy gradient method for RMDPs. In contrast with prior robust policy gradient algorithms, DRPG monotonically reduces approximation errors to guarantee convergence to a globally optimal policy in tabular RMDPs. We introduce a novel parametric transition kernel and solve the inner loop robust policy via a gradient-based method. Finally, our numerical results demonstrate the utility of our new algorithm and confirm its global convergence properties.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [(s, a)-rectangular Robust Policy Gradient Update on Randomly generated l1-robust MDPs](<https://www.sota2.com/research/sota/s-a-rectangular-robust-policy-gradient-update-on-randomly-generated-l1-robust-mdps.md>): (s, a)-rectangular Robust Policy Gradient Update; Randomly generated l1-robust MDPs; Relative Running Time: 326.

- [s-rectangular Robust Policy Gradient Update on Randomly generated l1-robust MDPs](<https://www.sota2.com/research/sota/s-rectangular-robust-policy-gradient-update-on-randomly-generated-l1-robust-mdps.md>): s-rectangular Robust Policy Gradient Update; Randomly generated l1-robust MDPs; Relative Running Time: 77.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2212.10439>)

[PDF](<https://arxiv.org/pdf/2212.10439.pdf>)
