Preserve Support, Not Correspondence: Dynamic Routing for Offline Reinforcement Learning

About

One-step offline RL actors are attractive because they avoid backpropagating through long iterative samplers and keep inference cheap, but they still have to improve under a critic without drifting away from actions that the dataset can support. In recent one-step extraction pipelines, a strong iterative teacher provides one target action for each latent draw, and the same student output is asked to do both jobs: move toward higher Q and stay near that paired endpoint. If those two directions disagree, the loss resolves them as a compromise on that same sample, even when a nearby better action remains locally supported by the data. We propose DROL, a latent-conditioned one-step actor trained with top-1 dynamic routing. For each state, the actor samples $K$ candidate actions from a bounded latent prior, assigns each dataset action to its nearest candidate, and updates only that winner with Behavior Cloning and critic guidance. Because the routing is recomputed from the current candidate geometry, ownership of a supported region can shift across candidates over the course of learning. This gives a one-step actor room to make local improvements that pointwise extraction struggles to capture, while retaining single-pass inference at test time. On OGBench and D4RL, DROL is competitive with the one-step FQL baseline, improving many OGBench task groups while remaining strong on both AntMaze and Adroit. Project page: https://muzhancun.github.io/preprints/DROL.

Zhancun Mu, Guangyu Zhao, Yiwu Zhong, Chi Zhang · 2026

Related benchmarks

Task	Dataset	Result
Offline Reinforcement Learning	OGBench antmaze-giant-navigate-singletask task1-v0 to task5-v0	Score45	33
Offline Reinforcement Learning	D4RL antmaze 6 tasks	Normalized Score81	21
Offline Reinforcement Learning	D4RL adroit (12 tasks)	Normalized Score52	21
Offline Reinforcement Learning	OGBench cube-single-singletask 5 tasks	Normalized Score98	14
Offline Reinforcement Learning	OGBench antmaze-large-singletask 5 tasks	Normalized Score90	11
Offline Reinforcement Learning	OGBench humanoidmaze-medium-singletask (5 tasks)	Normalized Score68	11
Offline Reinforcement Learning	OGBench scene-singletask 5 tasks	Normalized Score58	11
Offline Reinforcement Learning	OGBench humanoidmaze-large-singletask 5 tasks	Normalized Score11	11
Offline Reinforcement Learning	OGBench antsoccer-arena-singletask 5 tasks	Normalized Score65	11
Offline Reinforcement Learning	OGBench cube-double-singletask 5 tasks	Normalized Score26	11