Evolutionary Population Curriculum for Scaling Multi-Agent Reinforcement Learning

About

In multi-agent games, the complexity of the environment can grow exponentially as the number of agents increases, so it is particularly challenging to learn good policies when the agent population is large. In this paper, we introduce Evolutionary Population Curriculum (EPC), a curriculum learning paradigm that scales up Multi-Agent Reinforcement Learning (MARL) by progressively increasing the population of training agents in a stage-wise manner. Furthermore, EPC uses an evolutionary approach to fix an objective misalignment issue throughout the curriculum: agents successfully trained in an early stage with a small population are not necessarily the best candidates for adapting to later stages with scaled populations. Concretely, EPC maintains multiple sets of agents in each stage, performs mix-and-match and fine-tuning over these sets and promotes the sets of agents with the best adaptability to the next stage. We implement EPC on a popular MARL algorithm, MADDPG, and empirically show that our approach consistently outperforms baselines by a large margin as the number of agents grows exponentially.

Qian Long, Zihan Zhou, Abhibav Gupta, Fei Fang, Yi Wu, Xiaolong Wang · 2020

Related benchmarks

Task	Dataset	Result
Multi-Agent Reinforcement Learning	SMAC 3svs3z to 3svs4z	Environment Steps to 80% Reward1.29	7
Multi-Agent Reinforcement Learning	SMAC 3svs3z to 3svs5z	Environment Steps to 80% Reward2.18	7
Multi-Agent Reinforcement Learning	SMAC 8mvs8m to 8mvs9m	Steps to 80% Reward3.02	7
Multi-Agent Transfer Learning	SMAC 3svs3z to 3svs5z	Final Win Rate94	7
Multi-Agent Reinforcement Learning Transfer	SMAC 8m to 8mvs9m (transfer)	Steps to 80% Reward (x10^6)3.02	7
Multi-Agent Transfer Learning	SMAC 3svs3z to 3svs4z	Final Win Rate0.91	7
Multi-Agent Transfer Learning	SMAC 8mvs8m to 8mvs9m	Final Win Rate89	7
Coverage	Simple-Spread	Coverage Rate56.8	6