# Nash Learning from Human Feedback Source: [Nash Learning from Human Feedback]() Collection index: [llms.txt]() Data updated: 2025-10-26T02:53:13.696Z Authors: R\\'emi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland, Zhaohan Daniel Guo, Yunhao Tang, Matthieu Geist, Thomas Mesnard, Andrea Michi, Marco Selvi, Sertan Girgin, Nikola Momchev, Olivier Bachem, Daniel J. Mankowitz, Doina Precup, Bilal Piot Published: 2023-12-01T19:26:23.000Z ## Abstract Reinforcement learning from human feedback (RLHF) has emerged as the main paradigm for aligning large language models (LLMs) with human preferences. Typically, RLHF involves the initial step of learning a reward model from human feedback, often expressed as preferences between pairs of text generations produced by a pre-trained LLM. Subsequently, the LLM's policy is fine-tuned by optimizing it to maximize the reward model through a reinforcement learning algorithm. However, an inherent limitation of current reward models is their inability to fully represent the richness of human preferences and their dependency on the sampling distribution. In this study, we introduce an alternative pipeline for the fine-tuning of LLMs using pairwise human feedback. Our approach entails the initial learning of a preference model, which is conditioned on two inputs given a prompt, followed by the pursuit of a policy that consistently generates responses preferred over those generated by any competing policy, thus defining the Nash equilibrium of this preference model. We term this approach Nash learning from human feedback (NLHF). In the context of a tabular policy representation, we present a novel algorithmic solution, Nash-MD, founded on the principles of mirror descent. This algorithm produces a sequence of policies, with the last iteration converging to the regularized Nash equilibrium. Additionally, we explore parametric representations of policies and introduce gradient descent algorithms for deep-learning architectures. To demonstrate the effectiveness of our approach, we present experimental results involving the fine-tuning of a LLM for a text summarization task. We believe NLHF offers a compelling avenue for preference learning and policy optimization with the potential of advancing the field of aligning LLMs with human preferences. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Pairwise Preference Comparison on Qwen2.5-3B responses (test)](): Pairwise Preference Comparison; Qwen2.5-3B responses (test); Avg Preference Score: 82.7. - [Safety Alignment Robustness Evaluation on PKU-SafeRLHF (n=100 samples)](): Safety Alignment Robustness Evaluation; PKU-SafeRLHF (n=100 samples); Random Rate: 48.7. - [Preference Optimization on HelpSteer2 (test)](): Preference Optimization; HelpSteer2 (test); Avg Pref Score vs QWEN2.5-0.5B: 0.721. - [Pairwise Preference Comparison on Pairwise Preference Comparisons 1.5B Scale (test)](): Pairwise Preference Comparison; Pairwise Preference Comparisons 1.5B Scale (test); Avg Preference Score: 0.812. - [Combined Win Rate Evaluation on PKU-SafeRLHF Random prompts n = 100 samples](): Combined win rate evaluation; PKU-SafeRLHF Random prompts n = 100 samples; CVaR(0.125) Combined Win Rate: 22.9. - [CVaR(0.125) Combined Win Rate Evaluation on PKU-SafeRLHF (n=100 Conflict Prompts)](): Combined win rate evaluation; PKU-SafeRLHF Conflict prompts n = 100 samples; CVaR(0.125) Combined Win Rate: 28. - [Combined Win Rate Evaluation on PKU-SafeRLHF Sev-3 (n=100)](): Combined win rate evaluation; PKU-SafeRLHF Sev-3 prompts n = 100 samples; Combined Win Rate (CVaR 0.125): 52.5. - [Combined win rate evaluation on PKU-SafeRLHF prompts n = 100 samples (Sev-Low)](): Combined win rate evaluation; PKU-SafeRLHF prompts n = 100 samples (Sev-Low); CVaR(0.125) Combined Win Rate: 39.2. ## Original sources and implementation [Original paper]() [PDF]()