# Multi-Step First: A Lightweight Deep Reinforcement Learning Strategy for Robust Continuous Control with Partial Observability

Source: [Multi-Step First: A Lightweight Deep Reinforcement Learning Strategy for Robust Continuous Control with Partial Observability](<https://www.sota2.com/research/paper/68fa76981e209131cea6e04a>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-03-25T05:00:04.213Z

Authors: Lingheng Meng, Rob Gorbet, Michael Burke, Dana Kuli\\'c

Published: 2022-09-12T03:12:04.000Z

## Abstract

Deep Reinforcement Learning (DRL) has made considerable advances in simulated and physical robot control tasks, especially when problems admit a fully observed Markov Decision Process (MDP) formulation. When observations only partially capture the underlying state, the problem becomes a Partially Observable MDP (POMDP), and performance rankings between algorithms can change. We empirically compare Proximal Policy Optimization (PPO), Twin Delayed Deep Deterministic Policy Gradient (TD3), and Soft Actor-Critic (SAC) on representative POMDP variants of continuous-control benchmarks. Contrary to widely reported MDP results where TD3 and SAC typically outperform PPO, we observe an inversion: PPO attains higher robustness under partial observability. We attribute this to the stabilizing effect of multi-step bootstrapping. Furthermore, incorporating multi-step targets into TD3 (MTD3) and SAC (MSAC) improves their robustness. These findings provide practical guidance for selecting and adapting DRL algorithms in partially observable settings without requiring new theoretical machinery.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Robotic Control on Ant P-FLK](<https://www.sota2.com/research/sota/robotic-control-on-ant-p-flk.md>): Robotic Control; Ant P-FLK; Maximum Average Return: 2922.37.

- [Robotic Control on Ant MDP](<https://www.sota2.com/research/sota/robotic-control-on-ant-mdp.md>): Robotic Control; Ant MDP; Max Average Return: 5000.62.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2209.04999>)

[PDF](<https://arxiv.org/pdf/2209.04999.pdf>)
