# Extragradient Preference Optimization (EGPO): Beyond Last-Iterate Convergence for Nash Learning from Human Feedback Source: [Extragradient Preference Optimization (EGPO): Beyond Last-Iterate Convergence for Nash Learning from Human Feedback]() Collection index: [llms.txt]() Data updated: 2025-10-27T06:56:50.936Z Authors: Runlong Zhou, Maryam Fazel, Simon S. Du Published: 2025-03-11T22:44:54.000Z ## Abstract Reinforcement learning from human feedback (RLHF) has become essential for improving language model capabilities, but traditional approaches rely on the assumption that human preferences follow a transitive Bradley-Terry model. This assumption fails to capture the non-transitive nature of populational human preferences. Nash learning from human feedback (NLHF), targeting non-transitive preferences, is a problem of computing the Nash equilibrium (NE) of the two-player constant-sum game defined by the human preference. We introduce Extragradient preference optimization (EGPO), a novel algorithm for NLHF achieving last-iterate linear convergence to the NE of KL-regularized games and polynomial convergence to the NE of original games, while being robust to noise. Unlike previous approaches that rely on nested optimization, we derive an equivalent implementation using gradients of an online variant of the identity preference optimization (IPO) loss, enabling more faithful implementation for neural networks. Our empirical evaluations demonstrate EGPO's superior performance over baseline methods when training for the same number of epochs, as measured by pairwise win-rates using the ground truth preference. These results validate both the theoretical strengths and practical advantages of EGPO for language model alignment with non-transitive human preferences. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Safety Alignment Robustness Evaluation on PKU-SafeRLHF (n=100 samples)](): Safety Alignment Robustness Evaluation; PKU-SafeRLHF (n=100 samples); Random Rate: 23.2. - [Combined Win Rate Evaluation on PKU-SafeRLHF Random prompts n = 100 samples](): Combined win rate evaluation; PKU-SafeRLHF Random prompts n = 100 samples; CVaR(0.125) Combined Win Rate: 27.9. - [Combined Win Rate Evaluation on PKU-SafeRLHF Sev-3 (n=100)](): Combined win rate evaluation; PKU-SafeRLHF Sev-3 prompts n = 100 samples; Combined Win Rate (CVaR 0.125): 54.9. - [CVaR(0.125) Combined Win Rate Evaluation on PKU-SafeRLHF (n=100 Conflict Prompts)](): Combined win rate evaluation; PKU-SafeRLHF Conflict prompts n = 100 samples; CVaR(0.125) Combined Win Rate: 32.4. - [Combined win rate evaluation on PKU-SafeRLHF prompts n = 100 samples (Sev-Low)](): Combined win rate evaluation; PKU-SafeRLHF prompts n = 100 samples (Sev-Low); CVaR(0.125) Combined Win Rate: 41.2. ## Original sources and implementation [Original paper]() [PDF]()