Combinatorial Testing for Deep Learning Systems

• DOI: 10.48550/arxiv.1806.07723
• Publication Date: 2018-01-01
• Authors: Lei Ma; Fuyuan Zhang; Minhui Xue; Bo Li; Yang Liu; Jianjun Zhao; Yadong Wang
• Keywords: Robustness; Robustness testing; System testing

Deep learning (DL) has achieved remarkable progress over the past decade and been widely applied to many safety-critical applications. However, robustness of DL systems recently receives great concerns, such as adversarial examples against computer vision systems, which could potentially result in severe consequences. Adopting testing techniques help evaluate a system therefore detect vulnerabilities at an early stage. The main challenge is that its runtime state space too large: if we view each neuron for DL, then often contains massive states, rendering almost impossible. For traditional software, combinatorial (CT) effective technique reduce while obtaining relatively high defect detection abilities. In this paper, perform exploratory study CT on systems. We adapt concept propose set coverage criteria well guided test generation technique. Our evaluation demonstrates provides promising avenue further pose several open questions interesting directions