Advancing Weight and Channel Sparsification with Enhanced Saliency

• DOI: 10.48550/arxiv.2502.03658
• Publication Date: 2025-02-05
• Authors: Xinglong Sun; Maying Shen; Hongxu Yin; Lei Mao; Pavlo Molchanov; José M. Álvarez
• Keywords: FOS: Computer and information sciences; Computer Science - Machine Learning; Computer Vision and Pattern Recognition (cs.CV); Computer Science - Computer Vision and Pattern Recognition; Machine Learning (cs.LG)

Pruning aims to accelerate and compress models by removing redundant parameters, identified specifically designed importance scores which are usually imperfect. This removal is irreversible, often leading subpar performance in pruned models. Dynamic sparse training, while attempting adjust structures during training for continual reassessment refinement, has several limitations including criterion inconsistency between pruning growth, unsuitability structured sparsity, short-sighted growth strategies. Our paper introduces an efficient, innovative paradigm enhance a given either unstructured or sparsity. method separates the model into active structure exploitation exploration space potential updates. During exploitation, we optimize structure, whereas exploration, reevaluate reintegrate parameters from through growing step consistently guided same criterion. To prepare briefly "reactivate" all train them few iterations keeping part frozen, offering preview of gains reintegrating these parameters. We show on various datasets configurations that existing even simple as magnitude can be enhanced with ours achieve state-of-the-art cost reductions. Notably, ImageNet ResNet50, achieves +1.3 increase Top-1 accuracy over prior art at 90% ERK Compared SOTA latency HALP, reduced its 70% attaining faster more accurate model.