# Continuously Discovering Novel Strategies via Reward-Switching Policy Optimization

Source: [Continuously Discovering Novel Strategies via Reward-Switching Policy Optimization](<https://www.sota2.com/research/paper/68fa6c7d1e209131cea34446>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T00:11:46.889Z

Authors: Zihan Zhou, Wei Fu, Bingliang Zhang, Yi Wu

Published: 2022-04-04T12:38:58.000Z

## Abstract

We present Reward-Switching Policy Optimization (RSPO), a paradigm to discover diverse strategies in complex RL environments by iteratively finding novel policies that are both locally optimal and sufficiently different from existing ones. To encourage the learning policy to consistently converge towards a previously undiscovered local optimum, RSPO switches between extrinsic and intrinsic rewards via a trajectory-based novelty measurement during the optimization process. When a sampled trajectory is sufficiently distinct, RSPO performs standard policy optimization with extrinsic rewards. For trajectories with high likelihood under existing policies, RSPO utilizes an intrinsic diversity reward to promote exploration. Experiments show that RSPO is able to discover a wide spectrum of strategies in a variety of domains, ranging from single-agent particle-world tasks and MuJoCo continuous control to multi-agent stag-hunt games and StarCraftII challenges.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Multi-Agent Reinforcement Learning on GRF 3v1 hard](<https://www.sota2.com/research/sota/multi-agent-reinforcement-learning-on-grf-3v1-hard.md>): Multi-Agent Reinforcement Learning; GRF 3v1 hard; Win Rate: 94.

- [Multi-Agent Reinforcement Learning on GRF CA](<https://www.sota2.com/research/sota/multi-agent-reinforcement-learning-on-grf-ca.md>): Multi-Agent Reinforcement Learning; GRF (CA); Win Rate: 76.

- [Multi-Agent Reinforcement Learning on SMAC 2c\_vs\_64zg](<https://www.sota2.com/research/sota/multi-agent-reinforcement-learning-on-smac-2c_vs_64zg.md>): Multi-Agent Reinforcement Learning; SMAC 2c\_vs\_64zg; State Entropy: 0.07.

- [State Entropy Estimation on GRF (3v1)](<https://www.sota2.com/research/sota/state-entropy-estimation-on-grf-3v1.md>): State Entropy Estimation; GRF 3v1; State Entropy: 0.011.

- [Locomotion Diversity Discovery on Humanoid visual input](<https://www.sota2.com/research/sota/locomotion-diversity-discovery-on-humanoid-visual-input.md>): Locomotion Diversity Discovery; Humanoid visual input; Diversity Score: 0.53.

- [Strategy Discovery on GRF (CA)](<https://www.sota2.com/research/sota/strategy-discovery-on-grf-ca.md>): Strategy Discovery; GRF (CA); Distinct Strategies: 2.

- [State Entropy Estimation on GRF (CA)](<https://www.sota2.com/research/sota/state-entropy-estimation-on-grf-ca.md>): State Entropy Estimation; GRF (CA); State Entropy: 0.034.

- [Strategy Discovery on GRF (corner)](<https://www.sota2.com/research/sota/strategy-discovery-on-grf-corner.md>): Strategy Discovery; GRF Corner; Distinct Strategies: 1.6.

- [Multi-Agent Reinforcement Learning on SMAC 2m1z](<https://www.sota2.com/research/sota/multi-agent-reinforcement-learning-on-smac-2m1z.md>): Multi-Agent Reinforcement Learning; SMAC 2m1z; State Entropy: 0.032.

- [Multi-Agent Reinforcement Learning on GRF Corner (Win Rate and Ball Passing Routes)](<https://www.sota2.com/research/sota/multi-agent-reinforcement-learning-on-grf-corner-win-rate-and-ball-passing-routes.md>): Multi-Agent Reinforcement Learning; GRF Corner; Win Rate: 23.

- [Robot Locomotion on Humanoid](<https://www.sota2.com/research/sota/robot-locomotion-on-humanoid.md>): Robot Locomotion; Humanoid; Cumulative Reward: 1455.

- [Multi-Agent Reinforcement Learning on SMAC 2c64zg](<https://www.sota2.com/research/sota/multi-agent-reinforcement-learning-on-smac-2c64zg.md>): Multi-Agent Reinforcement Learning; SMAC 2c64zg; Win Rate: 85.

- [Strategy Discovery on GRF 3v1](<https://www.sota2.com/research/sota/strategy-discovery-on-grf-3v1.md>): Strategy Discovery; GRF 3v1; Distinct Strategies: 2.3.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2204.02246>)

[PDF](<https://arxiv.org/pdf/2204.02246.pdf>)
