NMTSloth: understanding and testing efficiency degradation of neural machine translation systems

• DOI: 10.1145/3540250.3549102
• Publication Date: 2022-11-09T20:46:22Z
• Authors: Simin Chen; Cong Liu; Mirazul Haque; Zihe Song; Wei Yang
• Keywords: Robustness

Neural Machine Translation (NMT) systems have received much recent attention due to their human-level accuracy. While existing works mostly focus on either improving accuracy or testing robustness, the computation efficiency of NMT systems, which is paramount importance often vast translation demands and real-time requirements, has surprisingly little attention. In this paper, we make first attempt understand test potential robustness in state-of-the-art systems. By analyzing working mechanism implementation 1455 public-accessible observe a fundamental property that could be manipulated an adversarial manner reduce significantly. Our interesting observation output length determines instead input, where depends two factors: sufficiently large yet pessimistic pre-configured threshold controlling max number iterations runtime generated end sentence (EOS) token. key motivation generate inputs delay generation EOS such would go through enough satisfy threshold. We present NMTSloth, develops gradient-guided technique searches for minimal unnoticeable perturbation at character-level, token-level, structure-level, delays appearance forces these reach naturally-unreachable To demonstrate effectiveness conduct systematic evaluation three public-available systems: Google T5, AllenAI WMT14, Helsinki-NLP translators. Experimental results show NMTSloth can increase systems' response latency energy consumption by 85% 3153% 86% 3052%, respectively, perturbing just one character token input sentence. case study shows significantly affect battery power real-world mobile devices (i.e., drain more than 30 times normal inputs).