# Transformers are Meta-Reinforcement Learners

Source: [Transformers are Meta-Reinforcement Learners](<https://www.sota2.com/research/paper/68fa6cc71e209131cea35f1b>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T00:18:38.162Z

Authors: Luckeciano C. Melo

Published: 2022-06-14T06:21:13.000Z

## Abstract

The transformer architecture and variants presented remarkable success across many machine learning tasks in recent years. This success is intrinsically related to the capability of handling long sequences and the presence of context-dependent weights from the attention mechanism. We argue that these capabilities suit the central role of a Meta-Reinforcement Learning algorithm. Indeed, a meta-RL agent needs to infer the task from a sequence of trajectories. Furthermore, it requires a fast adaptation strategy to adapt its policy for a new task -- which can be achieved using the self-attention mechanism. In this work, we present TrMRL (Transformers for Meta-Reinforcement Learning), a meta-RL agent that mimics the memory reinstatement mechanism using the transformer architecture. It associates the recent past of working memories to build an episodic memory recursively through the transformer layers. We show that the self-attention computes a consensus representation that minimizes the Bayes Risk at each layer and provides meaningful features to compute the best actions. We conducted experiments in high-dimensional continuous control environments for locomotion and dexterous manipulation. Results show that TrMRL presents comparable or superior asymptotic performance, sample efficiency, and out-of-distribution generalization compared to the baselines in these environments.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Meta Reinforcement Learning on MetaWorld ML45](<https://www.sota2.com/research/sota/meta-reinforcement-learning-on-metaworld-ml45.md>): Meta-Reinforcement Learning; MetaWorld ML45; Success Rate: 23.

- [Multi-task Reinforcement Learning on Meta-World MT10 v1 (Fixed)](<https://www.sota2.com/research/sota/multi-task-reinforcement-learning-on-meta-world-mt10-v1-fixed.md>): Multi-task reinforcement learning; Meta-World MT10 v1 (Fixed); Success Rate: 14.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2206.06614>)

[PDF](<https://arxiv.org/pdf/2206.06614.pdf>)
