Environment Design for Inverse Reinforcement Learning

About

Learning a reward function from demonstrations suffers from low sample-efficiency. Even with abundant data, current inverse reinforcement learning methods that focus on learning from a single environment can fail to handle slight changes in the environment dynamics. We tackle these challenges through adaptive environment design. In our framework, the learner repeatedly interacts with the expert, with the former selecting environments to identify the reward function as quickly as possible from the expert's demonstrations in said environments. This results in improvements in both sample-efficiency and robustness, as we show experimentally, for both exact and approximate inference.

Thomas Kleine Buening, Victor Villin, Christos Dimitrakakis · 2022

Related benchmarks

Task	Dataset	Result
Inverse Reinforcement Learning	MuJoCo Ant	Normalized Return0.82	7
Inverse Reinforcement Learning	MuJoCo Half-Cheetah	Normalized Return0.85	7
Inverse Reinforcement Learning	MuJoCo Walker	Normalized Return0.86	7
Inverse Reinforcement Learning	Half Cheetah (Target)	Mean Cumulative Reward4963.55	6
Inverse Reinforcement Learning	Ant Leg 0,2 disabled (Target)	Mean Cumulative Reward2190.12	6
Inverse Reinforcement Learning	Ant Leg 0,3 disabled (Source)	Mean Cumulative Rewards2231.09	6
Inverse Reinforcement Learning	Ant Leg 1,3 disabled (Target)	Mean Cumulative Reward2098.11	6
Inverse Reinforcement Learning	HalfCheetah rear disabled (Source)	Mean Cumulative Reward4014.52	6
Inverse Reinforcement Learning	HalfCheetah front disabled (Source)	Mean Cumulative Reward3905.38	6
Inverse Reinforcement Learning	HalfCheetah no disability (Target)	Mean Cumulative Reward3725.63	6