# Revisiting Self-Play Preference Optimization: On the Role of Prompt Difficulty Source: [Revisiting Self-Play Preference Optimization: On the Role of Prompt Difficulty]() Collection index: [llms.txt]() Data updated: 2026-03-04T05:00:12.633Z Authors: Yao Xiao, Jung-jae Kim, Roy Ka-wei Lee, Lidong Bing Published: 2025-10-07T02:47:25.000Z ## Abstract Self-play preference optimization has emerged as a prominent paradigm for aligning large language models (LLMs). It typically involves a language model to generate on-policy responses for prompts and a reward model (RM) to guide the selection of chosen and rejected responses, which can be further trained with direct preference optimization (DPO). However, the role of prompts remains underexplored, despite being a core component in this pipeline. In this work, we investigate how prompts of varying difficulty influence self-play preference optimization. We use the mean reward of sampled responses of a prompt as a proxy for its difficulty. We first find that difficult prompts exhibit substantially inferior self-play optimization performance compared to easy prompts for language models. Moreover, incorporating difficult prompts into training fails to enhance overall performance and, in fact, leads to slight degradation compared to training on easy prompts alone. Third, there is a clear upward trend in optimization performance as prompt difficulty decreases. We also observe that the performance gap between difficult and easy prompts tends to close as the model capacity increases, suggesting that prompt difficulty interacts with the model capacity. Building on these findings, we explore strategies to mitigate the adversary effect of difficult prompts on final performance. We demonstrate that only training on a small portion (30%) of the easiest prompts improves overall self-play performance on AlpacaEval~2 and Arena-Hard. We also report failed attempts and lessons learned. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [LLM Chat Evaluation on Arena-Hard v0.1 (test)](): LLM Chat Evaluation; Arena-Hard v0.1 (test); Win Rate: 40.1. - [Instruction Following Evaluation on AlpacaEval 2.0 (test)](): Instruction Following Evaluation; AlpacaEval 2.0 (test); LC% over π0: 38.89. ## Original sources and implementation [Original paper]() [PDF]()