Rethinking Optimal Transport in Offline Reinforcement Learning

About

We propose a novel algorithm for offline reinforcement learning using optimal transport. Typically, in offline reinforcement learning, the data is provided by various experts and some of them can be sub-optimal. To extract an efficient policy, it is necessary to \emph{stitch} the best behaviors from the dataset. To address this problem, we rethink offline reinforcement learning as an optimal transportation problem. And based on this, we present an algorithm that aims to find a policy that maps states to a \emph{partial} distribution of the best expert actions for each given state. We evaluate the performance of our algorithm on continuous control problems from the D4RL suite and demonstrate improvements over existing methods.

Arip Asadulaev, Rostislav Korst, Alexander Korotin, Vage Egiazarian, Andrey Filchenkov, Evgeny Burnaev · 2024

Related benchmarks

Task	Dataset	Result
Offline Reinforcement Learning	MuJoCo walker2d-medium D4RL	Normalized Return—	26
Offline Reinforcement Learning	MuJoCo halfcheetah-medium D4RL	Normalized Return—	20
Offline Reinforcement Learning	MuJoCo halfcheetah-medium-replay D4RL	Normalized Return—	20
Offline Reinforcement Learning	MuJoCo walker2d medium-replay D4RL	Normalized Return—	20
Offline Reinforcement Learning	MuJoCo halfcheetah-medium-expert D4RL	Normalized Return—	18
Offline Reinforcement Learning	MuJoCo walker2d medium-expert D4RL	Normalized Return—	18
Offline Reinforcement Learning	Android Hammer v2 (Human)	Score130.2	13
Offline Reinforcement Learning	Android Door Human v2	Normalized Score1.0466	13
Offline Reinforcement Learning	D4RL Adroit hammer-cloned v0	Normalized Score4.74	12
Offline Reinforcement Learning	MuJoCo Hopper Medium-Replay v2	Avg Normalized Score100.02	8