# Personalized Group Relative Policy Optimization for Heterogenous Preference Alignment Source: [Personalized Group Relative Policy Optimization for Heterogenous Preference Alignment]() Collection index: [llms.txt]() Data updated: 2026-03-13T05:00:04.336Z Authors: Jialu Wang, Heinrich Peters, Asad A. Butt, Navid Hashemi, Alireza Hashemi, Pouya M. Ghari, Joseph Hoover, James Rae, Morteza Dehghani Published: 2026-02-17T19:00:43.000Z ## Abstract Despite their sophisticated general-purpose capabilities, Large Language Models (LLMs) often fail to align with diverse individual preferences because standard post-training methods, like Reinforcement Learning with Human Feedback (RLHF), optimize for a single, global objective. While Group Relative Policy Optimization (GRPO) is a widely adopted on-policy reinforcement learning framework, its group-based normalization implicitly assumes that all samples are exchangeable, inheriting this limitation in personalized settings. This assumption conflates distinct user reward distributions and systematically biases learning toward dominant preferences while suppressing minority signals. To address this, we introduce Personalized GRPO (P-GRPO), a novel alignment framework that decouples advantage estimation from immediate batch statistics. By normalizing advantages against preference-group-specific reward histories rather than the concurrent generation group, P-GRPO preserves the contrastive signal necessary for learning distinct preferences. We evaluate P-GRPO across diverse tasks and find that it consistently achieves faster convergence and higher rewards than standard GRPO, thereby enhancing its ability to recover and align with heterogeneous preference signals. Our results demonstrate that accounting for reward heterogeneity at the optimization level is essential for building models that faithfully align with diverse human preferences without sacrificing general capabilities. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Language Generation on Synthetic Data (test)](): Language Generation; Synthetic data (test); ROUGE-1: 62.67. - [Language Generation on Goodreads Books (test)](): Language Generation; Goodreads Books (test); ROUGE-1: 61.38. - [Language Generation on KGRec-Music (test)](): Language Generation; KGRec-Music (test); Rouge-1: 56.18. - [Language Generation on KGRec (test)](): Language Generation; KGRec (test); ROUGE-1: 56.18. ## Original sources and implementation [Original paper]() [PDF]()