# Offline Constrained RLHF with Multiple Preference Oracles Source: [Offline Constrained RLHF with Multiple Preference Oracles]() Collection index: [llms.txt]() Data updated: 2026-04-03T05:00:04.568Z Authors: Brenden Latham, Mehrdad Moharrami Published: 2026-03-31T20:06:34.000Z ## Abstract We study offline constrained reinforcement learning from human feedback with multiple preference oracles. Motivated by applications that trade off performance with safety or fairness, we aim to maximize target population utility subject to a minimum protected group welfare constraint. From pairwise comparisons collected under a reference policy, we estimate oracle-specific rewards via maximum likelihood and analyze how statistical uncertainty propagates through the dual program. We cast the constrained objective as a KL-regularized Lagrangian whose primal optimizer is a Gibbs policy, reducing learning to a convex dual problem. We propose a dual-only algorithm that ensures high-probability constraint satisfaction and provide the first finite-sample performance guarantees for offline constrained preference learning. Finally, we extend our theoretical analysis to accommodate multiple constraints and general f-divergence regularization. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Offline Constrained RLHF on PKU-SafeRLHF 74k (train)](): Offline Constrained RLHF; PKU-SafeRLHF 74k (train); Safety Expected Reward (r1): -1.6034. - [Offline Constrained RLHF on PKU-SafeRLHF 7k (test)](): Offline Constrained RLHF; PKU-SafeRLHF 7k (test); Safety Reward (r1): -1.6044. ## Original sources and implementation [Original paper]() [PDF]()