Efficient Architecture Search by Network Transformation

• DOI: 10.1609/aaai.v32i1.11709
• Publication Date: 2022-06-24T21:08:34Z
• Authors: Han Cai; Tianyao Chen; Weinan Zhang; Yong Yu; Jun Wang
• Keywords: Benchmark (surveying); Scratch; Design space exploration

Techniques for automatically designing deep neural network architectures such as reinforcement learning based approaches have recently shown promising results. However, their success is on vast computational resources (e.g. hundreds of GPUs), making them difficult to be widely used. A noticeable limitation that they still design and train each from scratch during the exploration architecture space, which highly inefficient. In this paper, we propose a new framework toward efficient search by exploring space current reusing its weights. We employ agent meta-controller, whose action grow depth or layer width with function-preserving transformations. As such, previously validated networks can reused further exploration, thus saves large amount cost. apply our method explore plain convolutional (no skip-connections, branching etc.) image benchmark datasets (CIFAR-10, SVHN) restricted (5 GPUs). Our competitive outperform existing using same scheme. On CIFAR-10, model without skip-connections achieves 4.23% test error rate, exceeding majority modern approaching DenseNet. Furthermore, applying DenseNet are able achieve more accurate fewer parameters.