Offline RL Without Off-Policy Evaluation

About

Most prior approaches to offline reinforcement learning (RL) have taken an iterative actor-critic approach involving off-policy evaluation. In this paper we show that simply doing one step of constrained/regularized policy improvement using an on-policy Q estimate of the behavior policy performs surprisingly well. This one-step algorithm beats the previously reported results of iterative algorithms on a large portion of the D4RL benchmark. The one-step baseline achieves this strong performance while being notably simpler and more robust to hyperparameters than previously proposed iterative algorithms. We argue that the relatively poor performance of iterative approaches is a result of the high variance inherent in doing off-policy evaluation and magnified by the repeated optimization of policies against those estimates. In addition, we hypothesize that the strong performance of the one-step algorithm is due to a combination of favorable structure in the environment and behavior policy.

David Brandfonbrener, William F. Whitney, Rajesh Ranganath, Joan Bruna · 2021

Related benchmarks

Task	Dataset	Result
Offline Reinforcement Learning	D4RL halfcheetah-medium-expert	Normalized Score93.5	169
Offline Reinforcement Learning	D4RL hopper-medium-expert	Normalized Score102.1	161
Offline Reinforcement Learning	D4RL walker2d-medium-expert	Normalized Score112.9	140
Offline Reinforcement Learning	D4RL walker2d-random	Normalized Score6.1	101
Offline Reinforcement Learning	D4RL halfcheetah-random	Normalized Score6.9	94
Locomotion	D4RL walker2d-medium-expert	Normalized Score113	90
Offline Reinforcement Learning	D4RL hopper-random	Normalized Score7.8	86
Offline Reinforcement Learning	D4RL Walker2d Medium v2	Normalized Return81.8	85
Offline Reinforcement Learning	D4RL AntMaze	AntMaze Medium Play Return5	78
walker2d locomotion	D4RL walker2d medium-replay	Normalized Score49.5	78