Policy Evaluation on Reinforcement Learning (Theoretical Analysis)
-1Convergence RateGTD/GTD2
Evaluation results
| Method | Links | |
|---|---|---|
| GTD/GTD2Update Rule=yt = yt−1 + βt(µtbt − µt∆tθt−1 − Qtyt−1); θt = θt−1 + αt(µtA⊤t yt−1), Step-size schedule (alpha_t)=αt = βt = O(1/√t), Averaging/Projection Method=Projection+PR2017.09 | -1 | |
| GTD/GTD2Update Rule=yt = yt−1 + βt(µtbt − µt∆tθt−1 − Qtyt−1); θt = θt−1 + αt(µtA⊤t yt−1), Step-size schedule (alpha_t)=βt = ηαt, Σ αt = ∞, Σ αt^2 < ∞2017.09 | 0 | |
| TD(0)Update Rule=θt = θt−1 + αt(bt − ∆tθt−1), Step-size schedule (alpha_t)=αt = O(1/t)^β, β ∈ (0, 1), Averaging/Projection Method=PR-avg, Policy Protocol=on-policy2017.09 | 1 |