Diversity Actor-Critic: Sample-Aware Entropy Regularization for Sample-Efficient Exploration

About

In this paper, sample-aware policy entropy regularization is proposed to enhance the conventional policy entropy regularization for better exploration. Exploiting the sample distribution obtainable from the replay buffer, the proposed sample-aware entropy regularization maximizes the entropy of the weighted sum of the policy action distribution and the sample action distribution from the replay buffer for sample-efficient exploration. A practical algorithm named diversity actor-critic (DAC) is developed by applying policy iteration to the objective function with the proposed sample-aware entropy regularization. Numerical results show that DAC significantly outperforms existing recent algorithms for reinforcement learning.

Seungyul Han, Youngchul Sung · 2020

Related benchmarks

Task	Dataset	Result
Reinforcement Learning	Sparse MuJoCo Hopper v2 (test)	Max Average Return900.3	6
Reinforcement Learning	Delayed MuJoCo Hopper v2 (test)	Max Return3428.18	6
Reinforcement Learning	MuJoCo HalfCheetah Sparse v2 (test)	Max Average Return915.9	6
Reinforcement Learning	Delayed MuJoCo HalfCheetah v2 (test)	Average Return7594.7	6
Reinforcement Learning	Delayed MuJoCo Ant v2 (test)	Max Average Return4243.19	6
Reinforcement Learning	Sparse MuJoCo Ant v2 (test)	Max Average Return935.8	6
Reinforcement Learning	Sparse MuJoCo Walker2d v2 (test)	Max Return665.1	6
Reinforcement Learning	Delayed MuJoCo Walker2d v2 (test)	Max Average Return4067.11	6