# The Benefits of Being Distributional: Small-Loss Bounds for Reinforcement Learning Source: [The Benefits of Being Distributional: Small-Loss Bounds for Reinforcement Learning]() Collection index: [llms.txt]() Data updated: 2026-01-04T04:32:52.157Z Authors: Kaiwen Wang, Kevin Zhou, Runzhe Wu, Nathan Kallus, Wen Sun Published: 2023-05-25T04:19:43.000Z ## Abstract While distributional reinforcement learning (DistRL) has been empirically effective, the question of when and why it is better than vanilla, non-distributional RL has remained unanswered. This paper explains the benefits of DistRL through the lens of small-loss bounds, which are instance-dependent bounds that scale with optimal achievable cost. Particularly, our bounds converge much faster than those from non-distributional approaches if the optimal cost is small. As warmup, we propose a distributional contextual bandit (DistCB) algorithm, which we show enjoys small-loss regret bounds and empirically outperforms the state-of-the-art on three real-world tasks. In online RL, we propose a DistRL algorithm that constructs confidence sets using maximum likelihood estimation. We prove that our algorithm enjoys novel small-loss PAC bounds in low-rank MDPs. As part of our analysis, we introduce the $\\ell\_1$ distributional eluder dimension which may be of independent interest. Then, in offline RL, we show that pessimistic DistRL enjoys small-loss PAC bounds that are novel to the offline setting and are more robust to bad single-policy coverage. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Contextual Bandit on King County Housing (All episodes)](): Contextual Bandit; King County Housing (All episodes); Average Cost: 0.726. - [Contextual Bandit on King County Housing (Last 100 episodes)](): Contextual Bandit; King County Housing (Last 100 episodes); Average Cost: 0.708. - [Contextual Bandit on Prudential Life Insurance (All episodes)](): Contextual Bandit; Prudential Life Insurance (All episodes); Average Cost: 0.411. - [Contextual Bandit on Prudential Life Insurance (Last 100 episodes)](): Contextual Bandit; Prudential Life Insurance (Last 100 episodes); Average Cost: 0.388. - [Contextual Bandit on CIFAR-100 (All episodes)](): Contextual Bandit; CIFAR-100 (All episodes); Average Cost: 0.838. - [Contextual Bandit on CIFAR-100 (Last 100 episodes)](): Contextual Bandit; CIFAR-100 (Last 100 episodes); Average Cost: 0.775. ## Original sources and implementation [Original paper]() [PDF]() [Implementation]()