A Game-Theoretic Approach to Multi-Agent Trust Region Optimization

• DOI: 10.48550/arxiv.2106.06828
• Publication Date: 2023-01-01
• Authors: Wen, Ying; Chen, Hui; Yang, Yaodong; Tian, Zheng; Li, Minne; Chen, Xu; Wang, Jun
• Keywords: FOS: Computer and information sciences; Computer Science - Machine Learning; Artificial Intelligence (cs.AI); Computer Science - Artificial Intelligence; 0202 electrical engineering, electronic engineering, information engineering; Computer Science - Multiagent Systems; 02 engineering and technology; Multiagent Systems (cs.MA); Machine Learning (cs.LG)

A Multi-Agent Trust Region Learning (MATRL) algorithm that augments the single-agent trust region policy optimization with a weak stable fixed point approximated by the policy-space meta-game<br/>Trust region methods are widely applied in single-agent reinforcement learning problems due to their monotonic performance-improvement guarantee at every iteration. Nonetheless, when applied in multi-agent settings, the guarantee of trust region methods no longer holds because an agent's payoff is also affected by other agents' adaptive behaviors. To tackle this problem, we conduct a game-theoretical analysis in the policy space, and propose a multi-agent trust region learning method (MATRL), which enables trust region optimization for multi-agent learning. Specifically, MATRL finds a stable improvement direction that is guided by the solution concept of Nash equilibrium at the meta-game level. We derive the monotonic improvement guarantee in multi-agent settings and empirically show the local convergence of MATRL to stable fixed points in the two-player rotational differential game. To test our method, we evaluate MATRL in both discrete and continuous multiplayer general-sum games including checker and switch grid worlds, multi-agent MuJoCo, and Atari games. Results suggest that MATRL significantly outperforms strong multi-agent reinforcement learning baselines.<br/>