Towards Hyperparameter-free Policy Selection for Offline Reinforcement Learning

About

How to select between policies and value functions produced by different training algorithms in offline reinforcement learning (RL) -- which is crucial for hyperpa-rameter tuning -- is an important open question. Existing approaches based on off-policy evaluation (OPE) often require additional function approximation and hence hyperparameters, creating a chicken-and-egg situation. In this paper, we design hyperparameter-free algorithms for policy selection based on BVFT [XJ21], a recent theoretical advance in value-function selection, and demonstrate their effectiveness in discrete-action benchmarks such as Atari. To address performance degradation due to poor critics in continuous-action domains, we further combine BVFT with OPE to get the best of both worlds, and obtain a hyperparameter-tuning method for Q-function based OPE with theoretical guarantees as a side product.

Siyuan Zhang, Nan Jiang · 2021

Related benchmarks

Task	Dataset	Result
Offline Reinforcement Learning	D4RL Hopper medium	Reward1227.81	35
Offline Reinforcement Learning	Robomimic Can-Paired	Avg True Return71	7
Offline Reinforcement Learning	Robomimic Transport	Average True Return0.21	7
Offline Reinforcement Learning	Sepsis MDP n=200	Average True Return-19.32	6
Offline Reinforcement Learning	Sepsis POMDP (n=1000)	Average True Return-1.92	6