Decision Transformer: Reinforcement Learning via Sequence Modeling
About
We introduce a framework that abstracts Reinforcement Learning (RL) as a sequence modeling problem. This allows us to draw upon the simplicity and scalability of the Transformer architecture, and associated advances in language modeling such as GPT-x and BERT. In particular, we present Decision Transformer, an architecture that casts the problem of RL as conditional sequence modeling. Unlike prior approaches to RL that fit value functions or compute policy gradients, Decision Transformer simply outputs the optimal actions by leveraging a causally masked Transformer. By conditioning an autoregressive model on the desired return (reward), past states, and actions, our Decision Transformer model can generate future actions that achieve the desired return. Despite its simplicity, Decision Transformer matches or exceeds the performance of state-of-the-art model-free offline RL baselines on Atari, OpenAI Gym, and Key-to-Door tasks.
Key insights
How does sequence modeling solve offline RL?
The paper reframes offline reinforcement learning as autoregressive sequence modeling. A GPT-style transformer predicts actions from returns-to-go, states, and past actions, bypassing Bellman backups and value-function bootstrapping.
What distinguishes it from prior offline RL?
Unlike TD-learning baselines needing value pessimism, Decision Transformer directly outputs actions. On 1% DQN-replay Atari it matches CQL on 3 of 4 games and outperforms REM, QR-DQN, and BC on all four.
Where is the approach practically useful?
It is evaluated as a research prototype on Atari, OpenAI Gym D4RL, and Key-to-Door benchmarks, showing strong long-term credit assignment and robustness to sparse delayed rewards.
What limitations do the authors note?
The authors flag understanding transformer errors in MDP settings and dataset bias as underexplored. They also suggest future work on self-supervised pretraining, return-distribution conditioning, and trajectory modeling for model-based RL.