Uncertainty-Aware Action Advising for Deep Reinforcement Learning Agents

• DOI: 10.1609/aaai.v34i04.6036
• Publication Date: 2020-06-29T20:56:40Z
• Authors: Felipe Leno Da Silva; Pablo Hernandez-Leal; Bilal Kartal; Matthew E. Taylor
• Keywords: Leverage (statistics); Advice (programming); Sample (material); Sample complexity

Although Reinforcement Learning (RL) has been one of the most successful approaches for learning in sequential decision making problems, sample-complexity RL techniques still represents a major challenge practical applications. To combat this challenge, whenever competent policy (e.g., either legacy system or human demonstrator) is available, agent could leverage samples from (advice) to improve sample-efficiency. However, advice normally limited, hence it should ideally be directed states where uncertain on best action execute. In work, we propose Requesting Confidence-Moderated Policy (RCMP), an action-advising framework asks when its epistemic uncertainty high certain state. RCMP takes into account that limited and might suboptimal. We also describe technique estimate by performing minor modifications standard value-function-based methods. Our empirical evaluations show performs better than Importance Advising, not receiving advice, at random Gridworld Atari Pong scenarios.