A Unified Framework for Factorizing Distributional Value Functions for Multi-Agent Reinforcement Learning

About

In fully cooperative multi-agent reinforcement learning (MARL) settings, environments are highly stochastic due to the partial observability of each agent and the continuously changing policies of other agents. To address the above issues, we proposed a unified framework, called DFAC, for integrating distributional RL with value function factorization methods. This framework generalizes expected value function factorization methods to enable the factorization of return distributions. To validate DFAC, we first demonstrate its ability to factorize the value functions of a simple matrix game with stochastic rewards. Then, we perform experiments on all Super Hard maps of the StarCraft Multi-Agent Challenge and six self-designed Ultra Hard maps, showing that DFAC is able to outperform a number of baselines.

Wei-Fang Sun, Cheng-Kuang Lee, Simon See, Chun-Yi Lee · 2023

Related benchmarks

Task	Dataset	Result
Multi-Agent Reinforcement Learning	SMAC 6h_vs_8z Super Hard (test)	Averaged Score19.4	8
Multi-Agent Reinforcement Learning	SMAC 3s5z_vs_3s6z Super Hard (test)	Averaged Score20.94	8
Multi-Agent Reinforcement Learning	SMAC MMM2 Super Hard (test)	Averaged Score20.9	8
Multi-Agent Reinforcement Learning	SMAC 27m_vs_30m Super Hard (test)	Averaged Score19.71	8
Multi-Agent Reinforcement Learning	SMAC corridor Super Hard (test)	Average Score20	8
Multi-Agent Reinforcement Learning	SMAC Super Hard (test)	6h_vs_8z Win Rate83.92	8
Multi-Agent Reinforcement Learning	SMAC Ultra Hard (test)	MMM2 7m2M1M vs 8m4M1M Result63.35	6
Multi-Agent Reinforcement Learning	SMAC 6h_vs_9z Ultra Hard (test)	Average Score16	6
Multi-Agent Reinforcement Learning	SMAC 3s5z_vs_4s6z Ultra Hard (test)	Averaged Score19.65	6
Multi-Agent Reinforcement Learning	SMAC MMM2 (7m2M1M vs 8m4M1M) Ultra Hard (test)	Averaged Score16.5	6