# RIME: Robust Preference-based Reinforcement Learning with Noisy Preferences

Source: [RIME: Robust Preference-based Reinforcement Learning with Noisy Preferences](<https://www.sota2.com/research/paper/68fa756e1e209131cea67da3>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T03:33:53.801Z

Authors: Jie Cheng, Gang Xiong, Xingyuan Dai, Qinghai Miao, Yisheng Lv, Fei-Yue Wang

Published: 2024-02-27T07:03:25.000Z

## Abstract

Preference-based Reinforcement Learning (PbRL) circumvents the need for reward engineering by harnessing human preferences as the reward signal. However, current PbRL methods excessively depend on high-quality feedback from domain experts, which results in a lack of robustness. In this paper, we present RIME, a robust PbRL algorithm for effective reward learning from noisy preferences. Our method utilizes a sample selection-based discriminator to dynamically filter out noise and ensure robust training. To counteract the cumulative error stemming from incorrect selection, we suggest a warm start for the reward model, which additionally bridges the performance gap during the transition from pre-training to online training in PbRL. Our experiments on robotic manipulation and locomotion tasks demonstrate that RIME significantly enhances the robustness of the state-of-the-art PbRL method. Code is available at https://github.com/CJReinforce/RIME\_ICML2024.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Navigation on AntMaze (v2 Play Configurations)](<https://www.sota2.com/research/sota/navigation-on-antmaze-v2-play-configurations.md>): Navigation; AntMaze; Success Rate (Large Play v2): 10.4.

- [Robotic Manipulation on MetaWorld (Button Press, Door Open, Drawer Close, Window Close, Sweep Into Success Rates)](<https://www.sota2.com/research/sota/robotic-manipulation-on-metaworld-button-press-door-open-drawer-close-window-close-sweep-into-success-rates.md>): Robotic Manipulation; MetaWorld; Button Press Success Rate: 63.8.

- [Locomotion on D4RL MuJoCo Tasks (Average across v2 Environments)](<https://www.sota2.com/research/sota/locomotion-on-d4rl-mujoco-tasks-average-across-v2-environments.md>): Locomotion; D4RL MuJoCo Tasks; Average D4RL Locomotion Score (v2): 67.3.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2402.17257>)

[PDF](<https://arxiv.org/pdf/2402.17257.pdf>)