# StARformer: Transformer with State-Action-Reward Representations for Visual Reinforcement Learning Source: [StARformer: Transformer with State-Action-Reward Representations for Visual Reinforcement Learning]() Collection index: [llms.txt]() Data updated: 2025-10-26T00:54:03.791Z Authors: Jinghuan Shang, Kumara Kahatapitiya, Xiang Li, Michael S. Ryoo Published: 2021-10-12T17:55:30.000Z ## Abstract Reinforcement Learning (RL) can be considered as a sequence modeling task: given a sequence of past state-action-reward experiences, an agent predicts a sequence of next actions. In this work, we propose State-Action-Reward Transformer (StARformer) for visual RL, which explicitly models short-term state-action-reward representations (StAR-representations), essentially introducing a Markovian-like inductive bias to improve long-term modeling. Our approach first extracts StAR-representations by self-attending image state patches, action, and reward tokens within a short temporal window. These are then combined with pure image state representations -- extracted as convolutional features, to perform self-attention over the whole sequence. Our experiments show that StARformer outperforms the state-of-the-art Transformer-based method on image-based Atari and DeepMind Control Suite benchmarks, in both offline-RL and imitation learning settings. StARformer is also more compliant with longer sequences of inputs. Our code is available at https://github.com/elicassion/StARformer. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Dexterous Manipulation on D4RL Adroit human cloned v2](): Dexterous Manipulation; D4RL Adroit human cloned v2; Performance (Human Expert): 77.9. - [Locomotion on D4RL MuJoCo Tasks (Average across v2 Environments)](): Locomotion; D4RL MuJoCo Tasks; Average D4RL Locomotion Score (v2): 66.2. ## Original sources and implementation [Original paper]() [PDF]()