# Safe Reinforcement Learning with Preference-based Constraint Inference Source: [Safe Reinforcement Learning with Preference-based Constraint Inference]() Collection index: [llms.txt]() Data updated: 2026-05-26T05:00:13.075Z Authors: Chenglin Li, Grant Ruan, Hua Geng Published: 2026-03-24T08:54:03.000Z ## Abstract Safe reinforcement learning (RL) is a standard paradigm for safety-critical decision making. However, real-world safety constraints can be complex, subjective, and even hard to explicitly specify. Existing works on constraint inference rely on restrictive assumptions or extensive expert demonstrations, which are not realistic in many real-world applications. How to cheaply and reliably learn these constraints is the major challenge we focus on in this study. While inferring constraints from human preferences offers a data-efficient alternative, we identify popular Bradley-Terry (BT) models fail to capture the asymmetric, heavy-tailed nature of safety costs, resulting in risk underestimation. It is still rare in the literature to understand the impacts of BT models on the downstream policy learning. To address the above knowledge gaps, we propose a novel approach namely Preference-based Constrained Reinforcement Learning (PbCRL). We introduce a novel dead zone mechanism into preference modeling and theoretically prove that it encourages heavy-tailed cost distributions, thereby achieving better constraint alignment. Additionally, we incorporate a Signal-to-Noise Ratio (SNR) loss to encourage exploration by cost variances, which is found to benefit policy learning. Further, two-stage training strategy is deployed to lower online labeling burdens while adaptively enhancing constraint satisfaction. Empirical results demonstrate that PbCRL achieves superior alignment with true safety requirements and outperforms state-of-the-art baselines in terms of safety and reward. Our work explores a promising and effective way for constraint inference in Safe RL, with great potential in various safety-critical applications. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Language Model Alignment on Safe RLHF](): Language Model Alignment; Safe RLHF; Win Rate (Helpfulness): 80.7. - [Autonomous Driving on Autonomous driving simulator Block](): Autonomous Driving; Autonomous driving simulator Block; Return: 36.4. - [Autonomous Driving on Autonomous driving simulator Lane](): Autonomous Driving; Autonomous driving simulator Lane; Return: 51.4. - [Navigation on Safety Gymnasium Goal (test)](): Navigation; Safety Gymnasium Goal (test); Return: 23.41. - [Navigation on Safety Gymnasium Push (test)](): Navigation; Safety Gymnasium Push (test); Return: 3.2. - [Navigation on Safety Gymnasium Button (test)](): Navigation; Safety Gymnasium Button (test); Return: 8.73. - [Safe Reinforcement Learning on Safety Gymnasium HalfCheetah (threshold=5)](): Safe Reinforcement Learning; Safety Gymnasium HalfCheetah, threshold=5; Return: 2367. - [Safe Reinforcement Learning on Safety Gymnasium Walker2d (threshold=5)](): Safe Reinforcement Learning; Safety Gymnasium Walker2d, threshold=5; Return: 2659. - [Safe Reinforcement Learning on Safety Gymnasium Humanoid (threshold=5)](): Safe Reinforcement Learning; Safety Gymnasium Humanoid threshold=5; Return: 5247. - [Safe Reinforcement Learning on Safety Gymnasium Goal (threshold=20)](): Safe Reinforcement Learning; Safety Gymnasium Goal, threshold=20; Return: 23.41. - [Safe Reinforcement Learning on Safety Gymnasium Push (threshold=35)](): Safe Reinforcement Learning; Safety Gymnasium Push, threshold=35; Return: 3.2. - [Safe Reinforcement Learning on Safety Gymnasium Button (threshold=40)](): Safe Reinforcement Learning; Safety Gymnasium Button, threshold=40; Return: 8.73. ## Original sources and implementation [Original paper]() [PDF]()