Learning Structured Communication for Multi-agent Reinforcement Learning

• DOI: 10.48550/arxiv.2002.04235
• Publication Date: 2022-08-26
• Authors: Junjie Sheng; Xiangfeng Wang; Bo Jin; Junchi Yan; Wenhao Li; Tsung-Hui Chang; Jun Wang; Hongyuan Zha
• Keywords: FOS: Computer and information sciences; 0301 basic medicine; Computer Science - Machine Learning; 03 medical and health sciences; Statistics - Machine Learning; 0202 electrical engineering, electronic engineering, information engineering; Machine Learning (stat.ML); 02 engineering and technology; Machine Learning (cs.LG)

This work explores the large-scale multi-agent communication mechanism under a multi-agent reinforcement learning (MARL) setting. We summarize the general categories of topology for communication structures in MARL literature, which are often manually specified. Then we propose a novel framework termed as Learning Structured Communication (LSC) by using a more flexible and efficient communication topology. Our framework allows for adaptive agent grouping to form different hierarchical formations over episodes, which is generated by an auxiliary task combined with a hierarchical routing protocol. Given each formed topology, a hierarchical graph neural network is learned to enable effective message information generation and propagation among inter- and intra-group communications. In contrast to existing communication mechanisms, our method has an explicit while learnable design for hierarchical communication. Experiments on challenging tasks show the proposed LSC enjoys high communication efficiency, scalability, and global cooperation capability.