# Model-Driven Policy Optimization in Differentiable Simulators via Stochastic Exploration

Source: [Model-Driven Policy Optimization in Differentiable Simulators via Stochastic Exploration](<https://www.sota2.com/research/paper/6a0162b0dbc836b87d823257>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-05-12T05:00:12.036Z

Authors: Yuval Aroosh, Ayal Taitler

Published: 2026-05-08T09:52:25.000Z

## Abstract

Differentiable planning enables gradient-based optimization of decision-making problems by leveraging differentiable models of system dynamics. However, in highly nonlinear and hybrid discrete-continuous domains, the resulting optimization landscapes are often ill-conditioned, with flat regions and sharp transitions that hinder effective optimization.   We propose Model-Driven Policy Optimization (MDPO), a framework that introduces stochastic exploration into differentiable planning by injecting noise into the action space during optimization. Leveraging access to the model, MDPO further adapts the noise magnitude based on gradient-derived sensitivity of the trajectory objective, yielding a time-dependent exploration profile. This enables improved exploration of the objective landscape and helps escape poor local optima via dynamic allocation of exploration across timesteps and iterations.   Experiments on benchmark domains demonstrate that MDPO consistently outperforms deterministic differentiable planning, including both the noise-free variant of our method and available state-of-the-art implementations, as well as model-free baselines such as PPO, significantly improving solution quality across challenging nonlinear and hybrid settings. We further analyze the evolution of the adaptive noise magnitude across both time steps and optimization iterations, providing insight into how exploration is allocated during learning.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Differentiable Planning on Reservoir instance 30](<https://www.sota2.com/research/sota/differentiable-planning-on-reservoir-instance-30.md>): Differentiable Planning; Reservoir instance 30; Mean Performance: -815510.27.

- [Differentiable Planning on HVAC instance 10](<https://www.sota2.com/research/sota/differentiable-planning-on-hvac-instance-10.md>): Differentiable Planning; HVAC instance 10; Mean Performance: -1371468.

- [Differentiable Planning on HVAC instance 20](<https://www.sota2.com/research/sota/differentiable-planning-on-hvac-instance-20.md>): Differentiable Planning; HVAC instance 20; Mean Performance: -2086875.93.

- [Differentiable Planning on PowerGen instance 7](<https://www.sota2.com/research/sota/differentiable-planning-on-powergen-instance-7.md>): Differentiable Planning; PowerGen instance 7; Mean Performance: 10076.

- [Differentiable Planning on PowerGen instance 11](<https://www.sota2.com/research/sota/differentiable-planning-on-powergen-instance-11.md>): Differentiable Planning; PowerGen instance 11; Mean Performance: 16890.15.

- [Differentiable Planning on Reservoir instance 15](<https://www.sota2.com/research/sota/differentiable-planning-on-reservoir-instance-15.md>): Differentiable Planning; Reservoir instance 15; Mean Performance: -246613.49.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2605.07520>)

[PDF](<https://arxiv.org/pdf/2605.07520.pdf>)