# APPA: Adaptive Preference Pluralistic Alignment for Fair Federated RLHF of LLMs

Source: [APPA: Adaptive Preference Pluralistic Alignment for Fair Federated RLHF of LLMs](<https://www.sota2.com/research/paper/69d48fdd7d0172140ec746fd>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-04-08T05:00:12.103Z

Authors: Mahmoud Srewa, Tianyu Zhao, Salma Elmalaki

Published: 2026-04-05T20:48:36.000Z

## Abstract

Aligning large language models (LLMs) with diverse human preferences requires pluralistic alignment, where a single model must respect the values of multiple distinct groups simultaneously. In federated reinforcement learning from human feedback (FedRLHF), these groups align a shared policy without centralizing preference data, which makes fair reward aggregation essential. Existing aggregation methods exhibit clear trade offs: average based aggregation systematically under aligns worst performing groups, while min aggregation prioritizes worst group performance at the cost of overall alignment. We propose APPA, an Adaptive Preference Pluralistic Alignment framework that dynamically reweights group level rewards based on historical alignment rewards. Our approach prioritizes under aligned groups without degrading well aligned ones, while requiring no access to raw preference data. Integrated into a proximal policy optimization (PPO) based FedRLHF pipeline and evaluated on GLOBALQA and OQA across three model families (Gemma 2 2B, Llama 3.2 3B, Qwen3 0.6B), APPA achieves strong fairness alignment trade offs, improving worst group alignment by up to 28% over average aggregation while maintaining higher overall alignment than min aggregation across most configurations.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Ordinal Preference Alignment on GLOBALQA](<https://www.sota2.com/research/sota/ordinal-preference-alignment-on-globalqa.md>): Ordinal Preference Alignment; GLOBALQA; FI Bor.: 0.8911.

- [Ordinal Preference Alignment on OQA](<https://www.sota2.com/research/sota/ordinal-preference-alignment-on-oqa.md>): Ordinal Preference Alignment; OQA; FI Bor.: 72.13.

- [Reward-wise QA Fairness and Alignment on GLOBALQA](<https://www.sota2.com/research/sota/reward-wise-qa-fairness-and-alignment-on-globalqa.md>): Reward-wise QA fairness and alignment; GLOBALQA; FI JS Score: 0.9996.

- [Reward-wise QA fairness and alignment on OQA](<https://www.sota2.com/research/sota/reward-wise-qa-fairness-and-alignment-on-oqa.md>): Reward-wise QA fairness and alignment; OQA; JS Divergence (FI): 0.9909.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2604.04261>)

[PDF](<https://arxiv.org/pdf/2604.04261.pdf>)
