A Max-Min Entropy Framework for Reinforcement Learning

About

In this paper, we propose a max-min entropy framework for reinforcement learning (RL) to overcome the limitation of the soft actor-critic (SAC) algorithm implementing the maximum entropy RL in model-free sample-based learning. Whereas the maximum entropy RL guides learning for policies to reach states with high entropy in the future, the proposed max-min entropy framework aims to learn to visit states with low entropy and maximize the entropy of these low-entropy states to promote better exploration. For general Markov decision processes (MDPs), an efficient algorithm is constructed under the proposed max-min entropy framework based on disentanglement of exploration and exploitation. Numerical results show that the proposed algorithm yields drastic performance improvement over the current state-of-the-art RL algorithms.

Seungyul Han, Youngchul Sung · 2021

Related benchmarks

Task	Dataset	Result
Reinforcement Learning	Sparse MuJoCo Hopper v2 (test)	Max Average Return902.5	6
Reinforcement Learning	MuJoCo HalfCheetah Sparse v2 (test)	Max Average Return924.9	6
Reinforcement Learning	Sparse MuJoCo Walker2d v2 (test)	Max Return886.6	6
Reinforcement Learning	Sparse MuJoCo Ant v2 (test)	Max Average Return973.6	6
Reinforcement Learning	Delayed MuJoCo Hopper v2 (test)	Max Return3435.28	6
Reinforcement Learning	Delayed MuJoCo HalfCheetah v2 (test)	Average Return8451.2	6
Reinforcement Learning	Delayed MuJoCo Walker2d v2 (test)	Max Average Return5274.89	6
Reinforcement Learning	Delayed MuJoCo Ant v2 (test)	Max Average Return4851.64	6
Reinforcement Learning	MuJoCo Humanoid-Standup v2 (test)	Max Average Return267734.03	6
Reinforcement Learning	MuJoCo Humanoid v2 (test)	Max Avg Return9080.54	6