# Effective Diversity in Population Based Reinforcement Learning

Source: [Effective Diversity in Population Based Reinforcement Learning](<https://www.sota2.com/research/paper/68fa68811e209131cea1c7d4>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-25T22:38:59.898Z

Authors: Jack Parker-Holder, Aldo Pacchiano, Krzysztof Choromanski, Stephen Roberts

Published: 2020-02-03T10:09:16.000Z

## Abstract

Exploration is a key problem in reinforcement learning, since agents can only learn from data they acquire in the environment. With that in mind, maintaining a population of agents is an attractive method, as it allows data be collected with a diverse set of behaviors. This behavioral diversity is often boosted via multi-objective loss functions. However, those approaches typically leverage mean field updates based on pairwise distances, which makes them susceptible to cycling behaviors and increased redundancy. In addition, explicitly boosting diversity often has a detrimental impact on optimizing already fruitful behaviors for rewards. As such, the reward-diversity trade off typically relies on heuristics. Finally, such methods require behavioral representations, often handcrafted and domain specific. In this paper, we introduce an approach to optimize all members of a population simultaneously. Rather than using pairwise distance, we measure the volume of the entire population in a behavioral manifold, defined by task-agnostic behavioral embeddings. In addition, our algorithm Diversity via Determinants (DvD), adapts the degree of diversity during training using online learning techniques. We introduce both evolutionary and gradient-based instantiations of DvD and show they effectively improve exploration without reducing performance when better exploration is not required.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Multi-Agent Reinforcement Learning on SMAC 2c64zg](<https://www.sota2.com/research/sota/multi-agent-reinforcement-learning-on-smac-2c64zg.md>): Multi-Agent Reinforcement Learning; SMAC 2c64zg; Win Rate: 100.

- [Robot Locomotion on Humanoid](<https://www.sota2.com/research/sota/robot-locomotion-on-humanoid.md>): Robot Locomotion; Humanoid; Cumulative Reward: 4498.

- [Multi-Agent Reinforcement Learning on SMAC 2c\_vs\_64zg](<https://www.sota2.com/research/sota/multi-agent-reinforcement-learning-on-smac-2c_vs_64zg.md>): Multi-Agent Reinforcement Learning; SMAC 2c\_vs\_64zg; State Entropy: 0.057.

- [Adaptability Evaluation on BipedalWalker friction variations (test)](<https://www.sota2.com/research/sota/adaptability-evaluation-on-bipedalwalker-friction-variations-test.md>): Adaptability Evaluation; BipedalWalker friction variations (test); AUC: 1204.03.

- [Adaptability Evaluation on BipedalWalker mass variations (test)](<https://www.sota2.com/research/sota/adaptability-evaluation-on-bipedalwalker-mass-variations-test.md>): Adaptability Evaluation; BipedalWalker mass variations (test); AUC: 113.68.

- [State Entropy Estimation on GRF (3v1)](<https://www.sota2.com/research/sota/state-entropy-estimation-on-grf-3v1.md>): State Entropy Estimation; GRF 3v1; State Entropy: 0.01.

- [Quality-Diversity on Swimmer](<https://www.sota2.com/research/sota/quality-diversity-on-swimmer.md>): Quality Diversity; Swimmer; GT QD Score: 0.22.

- [Strategy Discovery on GRF 3v1](<https://www.sota2.com/research/sota/strategy-discovery-on-grf-3v1.md>): Strategy Discovery; GRF 3v1; Distinct Strategies: 3.

- [Multi-Agent Reinforcement Learning on GRF 3v1 hard](<https://www.sota2.com/research/sota/multi-agent-reinforcement-learning-on-grf-3v1-hard.md>): Multi-Agent Reinforcement Learning; GRF 3v1 hard; Win Rate: 83.

- [Quality-Diversity on Ant](<https://www.sota2.com/research/sota/quality-diversity-on-ant.md>): Quality Diversity; Ant; GT QD Score: 0.29.

- [Quality-Diversity on HalfCheetah](<https://www.sota2.com/research/sota/quality-diversity-on-halfcheetah.md>): Quality Diversity; Halfcheetah; GT QD Score: 0.85.

- [Quality-Diversity on Hopper](<https://www.sota2.com/research/sota/quality-diversity-on-hopper.md>): Quality Diversity; Hopper; GT QD Score: 0.56.

- [Quality-Diversity on Walker2d](<https://www.sota2.com/research/sota/quality-diversity-on-walker2d.md>): Quality Diversity; Walker2D; GT QD Score: 0.61.

- [Quality-Diversity on BipedalWalker](<https://www.sota2.com/research/sota/quality-diversity-on-bipedalwalker.md>): Quality Diversity; BipedalWalker; GT QD Score: 0.09.

- [Multi-Agent Reinforcement Learning on SMAC 2m1z](<https://www.sota2.com/research/sota/multi-agent-reinforcement-learning-on-smac-2m1z.md>): Multi-Agent Reinforcement Learning; SMAC 2m1z; State Entropy: 0.03.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2002.00632>)

[PDF](<https://arxiv.org/pdf/2002.00632.pdf>)
