# Pairwise Preference Reward and Group-Based Diversity Enhancement for Superior Open-Ended Generation Source: [Pairwise Preference Reward and Group-Based Diversity Enhancement for Superior Open-Ended Generation]() Collection index: [llms.txt]() Data updated: 2026-05-20T05:00:20.897Z Authors: Guining Cao, Jiaxin Peng, Chu Zeng, Yu Zhao, Shuangyong Song, Yongxiang Published: 2026-05-18T10:31:34.000Z ## Abstract Current reinforcement learning(RL) methods are broadly applicable and powerful in verifiable settings where scalar rewards can be provided. However, in open-ended generation tasks, verifying the correctness of responses remains challenging, and training reward models incurs substantial computational and annotation costs. Moreover, reinforcement learning (RLVR) often leads to diversity collapse and produces stereotypical or rigid outputs, outcomes that are particularly undesirable in open-domain scenarios. We propose Pairwise Preference Reward and Group-based Diversity Enhancement (PPR-GDE), a RL method that is more suitable for open-ended generation. PPR-GDE does not require scalar rewards and incorporates group-level diversity into the reward signal, it preserves the comparative structure of subjective evaluation through a pairwise preference reward, mitigates judge position bias via repeated comparisons with swapped response order, and introduces a group-based diversity reward that explicitly encourages semantic dispersion within a response group, all of these reward signals are integrated into a unified group-relative policy optimization objective. We instantiate PPR-GDE on role-playing task, experiments show that PPR-GDE achieves a better alignment quality as well as expressive diversity than strong RL baselines. Further analysis shows that pairwise preference is critical for preference alignment in subjective perspective, while the diversity metric plays an essential role in achieving superior expressive diversity and broader semantic coverage. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Multi-Turn Consistency on Role-playing dataset](): Multi-Turn Consistency; Role-playing dataset; KE Score: 2.65. - [Role-Playing Ability on Role-playing](): Role-Playing Ability; Role-playing; CUS Score: 4.99. - [Response Diversity on Role-playing dataset](): Response Diversity; Role-playing dataset; Distinct-2: 61. ## Original sources and implementation [Original paper]() [PDF]()