# Asymmetric Advantage Modulation Calibrates Entropy Dynamics in RLVR

Source: [Asymmetric Advantage Modulation Calibrates Entropy Dynamics in RLVR](<https://www.sota2.com/research/paper/69d48fe67d0172140ec74782>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-05-14T05:00:05.130Z

Authors: Hengrui Gu, Xiaotian Han, Yujing Bian, Feiyi Wang, Kaixiong Zhou

Published: 2026-04-06T17:42:38.000Z

## Abstract

Reinforcement learning with verifiable rewards (RLVR) has substantially improved the reasoning ability of large language models (LLMs), but it often suffers from \\textit{restricted exploration}, where the policy rapidly concentrates on a narrow set of solutions. A common remedy is entropy regularization, which attempts to preserve exploration by increasing policy entropy. However, for LLM-RL, this intervention is highly sensitive to its coefficient, can introduce semantically weak uncertainty, and often yields limited accuracy gains. This motivates a more precise question: which entropy helps reasoning, and which entropy should be reduced? To study this, we parameterize the advantage estimator in Group Relative Policy Optimization (GRPO) into positive and negative outcome-conditioned channels and analyze their entropy dynamics. Our results show that positive-channel modulation raises \\textit{productive entropy} associated with successful reasoning trajectories, while negative-channel modulation removes \\textit{noisy entropy} associated with failed rollouts and reduces interference with correct paths. Guided by this channel-wise view, we propose \\textbf{AsymGRPO}, which decouples the modulation strengths of positive and negative advantages. This enables flexible control over how the model updates across prompt difficulty levels, allowing stronger reinforcement of rare successes on harder prompts or stronger suppression of residual failures on easier prompts without forcing the two channels to share the same modulation strength. Experiments on five mathematical reasoning benchmarks show that AsymGRPO outperforms strong RLVR baselines, with consistent gains across model backbones.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Mathematical Reasoning on AIME 2024 (Avg@10 Accuracy)](<https://www.sota2.com/research/sota/mathematical-reasoning-on-aime-2024-avg-at-10-accuracy.md>): Mathematical Reasoning; AIME 2024; Avg@10 Accuracy: 39.33.

- [Mathematical Reasoning on AIME 2025 (Avg@10 accuracy)](<https://www.sota2.com/research/sota/mathematical-reasoning-on-aime-2025-avg-at-10-accuracy.md>): Mathematical Reasoning; AIME 2025; Avg@10 Accuracy: 36.

- [Mathematical Reasoning on AMC 2023 (Avg@10 accuracy)](<https://www.sota2.com/research/sota/mathematical-reasoning-on-amc-2023-avg-at-10-accuracy.md>): Mathematical Reasoning; AMC 2023; Avg@10 Accuracy: 83.25.

- [Mathematical Reasoning on OlympiadBench (Avg@5 accuracy)](<https://www.sota2.com/research/sota/mathematical-reasoning-on-olympiadbench-avg-at-5-accuracy.md>): Mathematical Reasoning; OlympiadBench; Avg@5 Accuracy: 58.93.

- [Mathematical Reasoning on MATH-500 (val)](<https://www.sota2.com/research/sota/mathematical-reasoning-on-math-500-val.md>): Mathematical Reasoning; MATH-500 (val); Avg@5 Accuracy: 89.73.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2604.04894>)

[PDF](<https://arxiv.org/pdf/2604.04894.pdf>)
