# Model-Free Opponent Shaping

Source: [Model-Free Opponent Shaping](<https://www.sota2.com/research/paper/68fa6dde1e209131cea3c916>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T00:45:11.347Z

Authors: Chris Lu, Timon Willi, Christian Schroeder de Witt, Jakob Foerster

Published: 2022-05-03T12:20:14.000Z

## Abstract

In general-sum games, the interaction of self-interested learning agents commonly leads to collectively worst-case outcomes, such as defect-defect in the iterated prisoner's dilemma (IPD). To overcome this, some methods, such as Learning with Opponent-Learning Awareness (LOLA), shape their opponents' learning process. However, these methods are myopic since only a small number of steps can be anticipated, are asymmetric since they treat other agents as naive learners, and require the use of higher-order derivatives, which are calculated through white-box access to an opponent's differentiable learning algorithm. To address these issues, we propose Model-Free Opponent Shaping (M-FOS). M-FOS learns in a meta-game in which each meta-step is an episode of the underlying inner game. The meta-state consists of the inner policies, and the meta-policy produces a new inner policy to be used in the next episode. M-FOS then uses generic model-free optimisation methods to learn meta-policies that accomplish long-horizon opponent shaping. Empirically, M-FOS near-optimally exploits naive learners and other, more sophisticated algorithms from the literature. For example, to the best of our knowledge, it is the first method to learn the well-known Zero-Determinant (ZD) extortion strategy in the IPD. In the same settings, M-FOS leads to socially optimal outcomes under meta-self-play. Finally, we show that M-FOS can be scaled to high-dimensional settings.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Iterated Prisoner's Dilemma on IPD-Analytic (round robin)](<https://www.sota2.com/research/sota/iterated-prisoner-s-dilemma-on-ipd-analytic-round-robin.md>): Iterated Prisoner's Dilemma; IPD-Analytic (round robin); Average Reward per Episode: -0.65.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2205.01447>)

[PDF](<https://arxiv.org/pdf/2205.01447.pdf>)
