# Policy Gradient for Rectangular Robust Markov Decision Processes

Source: [Policy Gradient for Rectangular Robust Markov Decision Processes](<https://www.sota2.com/research/paper/68fa71601e209131cea50f60>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-01-02T04:29:04.509Z

Authors: Navdeep Kumar, Esther Derman, Matthieu Geist, Kfir Levy, Shie Mannor

Published: 2023-01-31T12:40:50.000Z

## Abstract

Policy gradient methods have become a standard for training reinforcement learning agents in a scalable and efficient manner. However, they do not account for transition uncertainty, whereas learning robust policies can be computationally expensive. In this paper, we introduce robust policy gradient (RPG), a policy-based method that efficiently solves rectangular robust Markov decision processes (MDPs). We provide a closed-form expression for the worst occupation measure. Incidentally, we find that the worst kernel is a rank-one perturbation of the nominal. Combining the worst occupation measure with a robust Q-value estimation yields an explicit form of the robust gradient. Our resulting RPG can be estimated from data with the same time complexity as its non-robust equivalent. Hence, it relieves the computational burden of convex optimization problems required for training robust policies by current policy gradient approaches.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [(s, a)-rectangular Robust Policy Gradient Update on Randomly generated l1-robust MDPs](<https://www.sota2.com/research/sota/s-a-rectangular-robust-policy-gradient-update-on-randomly-generated-l1-robust-mdps.md>): (s, a)-rectangular Robust Policy Gradient Update; Randomly generated l1-robust MDPs; Relative Running Time: 1.3.

- [s-rectangular Robust Policy Gradient Update on Randomly generated l1-robust MDPs](<https://www.sota2.com/research/sota/s-rectangular-robust-policy-gradient-update-on-randomly-generated-l1-robust-mdps.md>): s-rectangular Robust Policy Gradient Update; Randomly generated l1-robust MDPs; Relative Running Time: 1.3.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2301.13589>)

[PDF](<https://arxiv.org/pdf/2301.13589.pdf>)

[Implementation](<https://github.com/navdtech/rpg>)
