# UCPO: Uncertainty-Aware Policy Optimization

Source: [UCPO: Uncertainty-Aware Policy Optimization](<https://www.sota2.com/research/paper/69802f68a8b01e76cc8e14db>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-05-28T05:00:13.160Z

Authors: Xianzhou Zeng, Jing Huang, Chunmei Xie, Gongrui Nan, Siye Chen, Mengyu Lu, Weiqi Xiong, Qixuan Zhou, Junhao Zhang, Qiang Zhu, Yadong Li, Xingzhong Xu

Published: 2026-01-30T07:07:42.000Z

## Abstract

The key to building trustworthy large language models (LLMs) lies in endowing them with inherent uncertainty expression capabilities, thereby mitigating overconfident errors in high-stakes applications. However, existing RL paradigms such as GRPO often suffer from Advantage Bias due to binary decision spaces and static uncertainty rewards, inducing either excessive conservatism or overconfidence. To tackle this challenge, this paper unveils the root causes of reward hacking and overconfidence in current RL paradigms incorporating uncertainty-based rewards, based on which we propose the UnCertainty-Aware Policy Optimization (UCPO) framework. UCPO employs Ternary Advantage Decoupling to separate and independently normalize deterministic and uncertain rollouts, thereby eliminating advantage bias. Furthermore, a Dynamic Uncertainty Reward Adjustment mechanism adapts uncertainty weights in real-time according to model evolution and instance difficulty. Experimental results in mathematical reasoning and general tasks demonstrate that UCPO effectively resolves the reward imbalance, significantly improving the reliability of the model beyond their knowledge boundaries.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [General Tasks on GPQA Diamond](<https://www.sota2.com/research/sota/general-tasks-on-gpqa-diamond.md>): General Tasks; GPQA Diamond; PAQ Score: 0.677.

- [General Tasks on MMLU-Redux2](<https://www.sota2.com/research/sota/general-tasks-on-mmlu-redux2.md>): General Tasks; MMLU-Redux2; PAQ: 91.67.

- [Math and Text Reasoning on AIME 24](<https://www.sota2.com/research/sota/math-and-text-reasoning-on-aime-24.md>): Math and Text Reasoning; AIME 24; PAQ: 86.11.

- [Math and Text Reasoning on MATH 500](<https://www.sota2.com/research/sota/math-and-text-reasoning-on-math-500.md>): Math and Text Reasoning; MATH 500; PAQ: 97.28.

- [Math and Text Reasoning on Olympiad Bench](<https://www.sota2.com/research/sota/math-and-text-reasoning-on-olympiad-bench.md>): Math and Text Reasoning; Olympiad Bench; PAQ: 73.67.

- [Math and Text Reasoning on AMC (PAQ, F1)](<https://www.sota2.com/research/sota/math-and-text-reasoning-on-amc-paq-f1.md>): Math and Text Reasoning; AMC; PAQ: 91.95.

- [Math and Text Reasoning on Minerva](<https://www.sota2.com/research/sota/math-and-text-reasoning-on-minerva.md>): Math and Text Reasoning; Minerva; PAQ: 0.4915.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2601.22648>)

[PDF](<https://arxiv.org/pdf/2601.22648.pdf>)