# Unifying Stable Optimization and Reference Regularization in RLHF

Source: [Unifying Stable Optimization and Reference Regularization in RLHF](<https://www.sota2.com/research/paper/698eaff3c97a0b3e3b5e6617>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-02-14T05:00:04.349Z

Authors: Li He, Qiang Qu, He Zhao, Stephen Wan, Dadong Wang, Lina Yao, Tongliang Liu

Published: 2026-02-12T03:31:19.000Z

## Abstract

Reinforcement Learning from Human Feedback (RLHF) has advanced alignment capabilities significantly but remains hindered by two core challenges: \\textbf{reward hacking} and \\textbf{stable optimization}. Current solutions independently address these issues through separate regularization strategies, specifically a KL-divergence penalty against a supervised fine-tuned model ($\\pi\_0$) to mitigate reward hacking, and policy ratio clipping towards the current policy ($\\pi\_t$) to promote stable alignment. However, the implicit trade-off arising from simultaneously regularizing towards both $\\pi\_0$ and $\\pi\_t$ remains under-explored. In this paper, we introduce a unified regularization approach that explicitly balances the objectives of preventing reward hacking and maintaining stable policy updates. Our simple yet principled alignment objective yields a weighted supervised fine-tuning loss with a superior trade-off, which demonstrably improves both alignment results and implementation complexity. Extensive experiments across diverse benchmarks validate that our method consistently outperforms RLHF and online preference learning methods, achieving enhanced alignment performance and stability.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Instruction Following Evaluation on AlpacaEval 2.0 (test)](<https://www.sota2.com/research/sota/instruction-following-evaluation-on-alpacaeval-2-0-test.md>): Instruction Following Evaluation; AlpacaEval 2.0 (test); LC% over π0: 54.17.

- [Multi-turn Conversation Evaluation on MT-Bench 1.0 (test)](<https://www.sota2.com/research/sota/multi-turn-conversation-evaluation-on-mt-bench-1-0-test.md>): Multi-turn Conversation Evaluation; MT-Bench 1.0 (test); GPT-4 Score: 8.538.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2602.11523>)

[PDF](<https://arxiv.org/pdf/2602.11523.pdf>)