Trust Region Policy Optimization

About

We describe an iterative procedure for optimizing policies, with guaranteed monotonic improvement. By making several approximations to the theoretically-justified procedure, we develop a practical algorithm, called Trust Region Policy Optimization (TRPO). This algorithm is similar to natural policy gradient methods and is effective for optimizing large nonlinear policies such as neural networks. Our experiments demonstrate its robust performance on a wide variety of tasks: learning simulated robotic swimming, hopping, and walking gaits; and playing Atari games using images of the screen as input. Despite its approximations that deviate from the theory, TRPO tends to give monotonic improvement, with little tuning of hyperparameters.

John Schulman, Sergey Levine, Philipp Moritz, Michael I. Jordan, Pieter Abbeel · 2015

Related benchmarks

Task	Dataset	Result
Reinforcement Learning	Walker	Average Returns229.8	38
Reinforcement Learning	HalfCheetah v3	Mean Reward4785	34
Quadruped	Quadruped	Return206.24	33
Reinforcement Learning	Humanoid	Zero-Shot Reward3175	32
Reinforcement Learning	Hopper v4	Average Return2017	30
Reinforcement Learning	LunarLander v2	Final Return104	30
Reinforcement Learning	cartpole	Average Reward977.9	29
Optimal Policy Selection	Stochastic Gridworld (S=16, A=4)	PCS4	28
Reinforcement Learning	Pendulum	Avg Episode Reward-145.49	26
Reinforcement Learning	Ant v3	Average Final Return6203	26