Diffusion Model Predictive Control

About

We propose Diffusion Model Predictive Control (D-MPC), a novel MPC approach that learns a multi-step action proposal and a multi-step dynamics model, both using diffusion models, and combines them for use in online MPC. On the popular D4RL benchmark, we show performance that is significantly better than existing model-based offline planning methods using MPC (e.g. MBOP) and competitive with state-of-the-art (SOTA) model-based and model-free reinforcement learning methods. We additionally illustrate D-MPC's ability to optimize novel reward functions at run time and adapt to novel dynamics, and highlight its advantages compared to existing diffusion-based planning baselines.

Guangyao Zhou, Sivaramakrishnan Swaminathan, Rajkumar Vasudeva Raju, J. Swaroop Guntupalli, Wolfgang Lehrach, Joseph Ortiz, Antoine Dedieu, Miguel Lázaro-Gredilla, Kevin Murphy · 2024

Related benchmarks

Task	Dataset	Result
Offline Reinforcement Learning	D4RL Medium-Replay Hopper	Normalized Score92.5	109
Offline Reinforcement Learning	D4RL Medium HalfCheetah	Normalized Score46	105
Offline Reinforcement Learning	D4RL Medium Walker2d	Normalized Score76.2	104
Offline Reinforcement Learning	D4RL Medium-Replay HalfCheetah	Normalized Score41.1	97
Offline Reinforcement Learning	D4RL Medium Hopper	Normalized Score61.2	72
Offline Reinforcement Learning	D4RL Medium-Replay Walker2d	Normalized Score78.8	52
Offline Reinforcement Learning	scene-play OGBench 5 tasks v0	Average Success Rate4	33
Offline Reinforcement Learning	puzzle-4x4-play OGBench 5 tasks v0	Average Success Rate0	28
Offline Reinforcement Learning	cube-double-play OGBench 5 tasks v0	Average Success Rate3	19
Offline Reinforcement Learning	puzzle-3x3-play OGBench 5 tasks v0	Average Success Rate1	19