AutoReP: Automatic ReLU Replacement for Fast Private Network Inference

• DOI: 10.48550/arxiv.2308.10134
• Publication Date: 2023-01-01
• Authors: Hongwu Peng; Shaoyi Huang; Tong Zhou; Yukui Luo; Chenghong Wang; Zigeng Wang; Jiahui Zhao; Xie Xi; Ang Li; Tony Geng; Kaleel Mahmood; Wujie Wen; Xiaolin Xu; Caiwen Ding
• Keywords: I.2; FOS: Computer and information sciences; Computer Science - Machine Learning; Computer Science - Cryptography and Security; E.3; I.2; B.0; E.3; Cryptography and Security (cs.CR); B.0; Machine Learning (cs.LG)

The growth of the Machine-Learning-As-A-Service (MLaaS) market has highlighted clients' data privacy and security issues. Private inference (PI) techniques using cryptographic primitives offer a solution but often have high computation communication costs, particularly with non-linear operators like ReLU. Many attempts to reduce ReLU operations exist, they may need heuristic threshold selection or cause substantial accuracy loss. This work introduces AutoReP, gradient-based approach lessen alleviate these It automates polynomial functions speed up PI applications distribution-aware approximation (DaPa) maintain model expressivity while accurately approximating ReLUs. Our experimental results demonstrate significant improvements 6.12% (94.31%, 12.9K budget, CIFAR-10), 8.39% (74.92%, CIFAR-100), 9.45% (63.69%, 55K Tiny-ImageNet) over current state-of-the-art methods, e.g., SNL. Morever, AutoReP is applied EfficientNet-B2 on ImageNet dataset, achieved 75.55% 176.1 times budget reduction.