# Uncertainty-Aware LLM-Guided Policy Shaping for Sparse-Reward Reinforcement Learning Source: [Uncertainty-Aware LLM-Guided Policy Shaping for Sparse-Reward Reinforcement Learning]() Collection index: [llms.txt]() Data updated: 2026-06-09T05:00:07.006Z Authors: Ujjwal Bhatta, Utsabi Dangol, Sumaly Bajracharya, Rodrigue Rizk, KC Santosh Published: 2026-06-04T19:46:45.000Z ## Abstract Sparse rewards and heterogeneous task sequences remain persistent challenges in Reinforcement Learning (RL), often resulting in slow convergence, weak generalization, and inefficient exploration. We propose Uncertainty-Aware LLM-Guided Policy Shaping (ULPS), a novel framework that integrates a calibrated Large Language Model (LLM) into the RL training loop to provide structured, uncertainty-modulated behavioral guidance. ULPS employs an A\*-based oracle to synthesize optimal symbolic trajectories, which are used to fine-tune a BERT-based language model. During training, this model supplies action suggestions whose influence is conditioned on epistemic uncertainty estimated via Monte Carlo (MC) dropout. An entropy-based blending mechanism adaptively balances LLM guidance and the learned policy (via Proximal Policy Optimization, PPO), allowing the agent to prioritize reliable priors while preserving adaptability. We evaluate ULPS on the MiniGridUnlockPickup benchmark and observe consistent improvements in success rate, reward efficiency, and sample complexity over unguided, uncalibrated, and standard RL baselines. ULPS achieves more than 9% improvement in execution accuracy after fine-tuning, requires fewer environment interactions, and yields higher reward AUC. Our results demonstrate that integrating symbolic A\* trajectories, pretrained language priors, and uncertainty-aware control offers a principled and effective approach to multi-task reinforcement learning in sparse-reward domains, with potential extensibility to partially observable and multi-agent settings. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [UnlockPickup on MiniGrid 4x4 (test)](): UnlockPickUp; MiniGrid UnlockPickup 4x4 (test); Reward AUC: 2055.08. - [UnlockPickup on MiniGrid 8x8 (test)](): UnlockPickUp; MiniGrid UnlockPickup 8x8 (test); Reward AUC: 1886.8. - [Action Prediction on Mini-grid 4x4](): Action Prediction; Mini-grid 4x4; Accuracy: 99.17. - [Action Prediction on Mini-grid 3x3](): Action Prediction; Mini-grid 3x3; Accuracy: not recorded. ## Original sources and implementation [Original paper]() [PDF]()