Anti-Exploration by Random Network Distillation

About

Despite the success of Random Network Distillation (RND) in various domains, it was shown as not discriminative enough to be used as an uncertainty estimator for penalizing out-of-distribution actions in offline reinforcement learning. In this paper, we revisit these results and show that, with a naive choice of conditioning for the RND prior, it becomes infeasible for the actor to effectively minimize the anti-exploration bonus and discriminativity is not an issue. We show that this limitation can be avoided with conditioning based on Feature-wise Linear Modulation (FiLM), resulting in a simple and efficient ensemble-free algorithm based on Soft Actor-Critic. We evaluate it on the D4RL benchmark, showing that it is capable of achieving performance comparable to ensemble-based methods and outperforming ensemble-free approaches by a wide margin.

Alexander Nikulin, Vladislav Kurenkov, Denis Tarasov, Sergey Kolesnikov · 2023

Related benchmarks

Task	Dataset	Result
Offline Reinforcement Learning	D4RL antmaze-umaze (diverse)	Normalized Score66	74
Offline Reinforcement Learning	hopper medium	Normalized Score97.8	68
Offline Reinforcement Learning	walker2d medium-replay	Normalized Score88.7	61
Offline Reinforcement Learning	walker2d medium	Normalized Score91.6	61
Offline Reinforcement Learning	hopper medium-replay	Normalized Score100.5	55
Offline Reinforcement Learning	halfcheetah medium-replay	Normalized Score54.9	54
Offline Reinforcement Learning	halfcheetah medium	Normalized Score66.6	53
Offline Reinforcement Learning	D4RL Adroit pen (human)	Normalized Return5.6	53
Offline Reinforcement Learning	D4RL Adroit pen (cloned)	Normalized Return2.5	53
Offline Reinforcement Learning	D4RL antmaze-large (play)	Normalized Score43.9	47