# Model-Free Reinforcement Learning with the Decision-Estimation Coefficient

Source: [Model-Free Reinforcement Learning with the Decision-Estimation Coefficient](<https://www.sota2.com/research/paper/68fa5aca1e209131ce9c2c4d>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-12-24T08:15:46.476Z

Authors: Dylan J. Foster, Noah Golowich, Jian Qian, Alexander Rakhlin, Ayush Sekhari

Published: 2022-11-25T17:29:40.000Z

## Abstract

We consider the problem of interactive decision making, encompassing structured bandits and reinforcement learning with general function approximation. Recently, Foster et al. (2021) introduced the Decision-Estimation Coefficient, a measure of statistical complexity that lower bounds the optimal regret for interactive decision making, as well as a meta-algorithm, Estimation-to-Decisions, which achieves upper bounds in terms of the same quantity. Estimation-to-Decisions is a reduction, which lifts algorithms for (supervised) online estimation into algorithms for decision making. In this paper, we show that by combining Estimation-to-Decisions with a specialized form of optimistic estimation introduced by Zhang (2022), it is possible to obtain guarantees that improve upon those of Foster et al. (2021) by accommodating more lenient notions of estimation error. We use this approach to derive regret bounds for model-free reinforcement learning with value function approximation, and give structural results showing when it can and cannot help more generally.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Model-free learning on Stochastic MDPs Bilinear or BE](<https://www.sota2.com/research/sota/model-free-learning-on-stochastic-mdps-bilinear-or-be.md>): Model-free learning; Stochastic MDPs Bilinear or BE; Regret T Exponent: 3.

- [Regret Bound Estimation on Hybrid MDPs (Bilinear or Coverable, Bellman Complete, On-Policy)](<https://www.sota2.com/research/sota/regret-bound-estimation-on-hybrid-mdps-bilinear-or-coverable-bellman-complete-on-policy.md>): Regret Bound Estimation; Hybrid MDPs {Bilinear or Coverable} + Bellman Complete + On-Policy setting; Regret Bound (On-Policy): not recorded.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2211.14250>)

[PDF](<https://arxiv.org/pdf/2211.14250.pdf>)
