# PrefMMT: Modeling Human Preferences in Preference-based Reinforcement Learning with Multimodal Transformers Source: [PrefMMT: Modeling Human Preferences in Preference-based Reinforcement Learning with Multimodal Transformers]() Collection index: [llms.txt]() Data updated: 2025-10-26T04:15:51.701Z Authors: Dezhong Zhao, Ruiqi Wang, Dayoon Suh, Taehyeon Kim, Ziqin Yuan, Byung-Cheol Min, Guohua Chen Published: 2024-09-20T17:51:24.000Z ## Abstract Preference-based reinforcement learning (PbRL) shows promise in aligning robot behaviors with human preferences, but its success depends heavily on the accurate modeling of human preferences through reward models. Most methods adopt Markovian assumptions for preference modeling (PM), which overlook the temporal dependencies within robot behavior trajectories that impact human evaluations. While recent works have utilized sequence modeling to mitigate this by learning sequential non-Markovian rewards, they ignore the multimodal nature of robot trajectories, which consist of elements from two distinctive modalities: state and action. As a result, they often struggle to capture the complex interplay between these modalities that significantly shapes human preferences. In this paper, we propose a multimodal sequence modeling approach for PM by disentangling state and action modalities. We introduce a multimodal transformer network, named PrefMMT, which hierarchically leverages intra-modal temporal dependencies and inter-modal state-action interactions to capture complex preference patterns. We demonstrate that PrefMMT consistently outperforms state-of-the-art PM baselines on locomotion tasks from the D4RL benchmark and manipulation tasks from the Meta-World benchmark. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Navigation on AntMaze (v2 Play Configurations)](): Navigation; AntMaze; Success Rate (Large Play v2): 36.8. - [Robotic Manipulation on MetaWorld (Button Press, Door Open, Drawer Close, Window Close, Sweep Into Success Rates)](): Robotic Manipulation; MetaWorld; Button Press Success Rate: 78.6. - [Locomotion on D4RL MuJoCo Tasks (Average across v2 Environments)](): Locomotion; D4RL MuJoCo Tasks; Average D4RL Locomotion Score (v2): 77.4. ## Original sources and implementation [Original paper]() [PDF]()