A Hybrid CNN-Transformer Architecture with Frequency Domain Contrastive Learning for Image Deraining

• DOI: 10.48550/arxiv.2308.03340
• Publication Date: 2023-08-11
• Authors: Wang, Cheng; Li, Wei
• Keywords: FOS: Computer and information sciences; I.4.4; Computer Vision and Pattern Recognition (cs.CV); Image and Video Processing (eess.IV); Computer Science - Computer Vision and Pattern Recognition; FOS: Electrical engineering, electronic engineering, information engineering; 0211 other engineering and technologies; 02 engineering and technology; Electrical Engineering and Systems Science - Image and Video Processing

Abstract Image deraining is a challenging task that involves restoring degraded images affected by rain streaks. While Convolutional Neural Networks (CNNs) have been commonly used for this task, existing approaches often rely on stacked convolutional basic blocks with limited performance and compromised spatial detail. Furthermore, the limited receptive field of convolutional layers leads to incomplete processing of non-uniform rain streaks. To address these concerns, we propose a novel image deraining network that combines CNNs and trans- formers. Our network comprises two stages: an encoder-decoder architecture with a triple attention mechanism to capture valuable features and residual dual branch transformer blocks that enhance local information modeling. To address the transformer’s lack of local information modeling capability, we intro- duce convolution in the self-attentive mechanism of the transformer block and feed-forward network. Additionally, we employ a frequency domain contrastive learning method to enhance contrastive sample information, ensuring that the restored image closely resembles the clear image in the frequency domain space, while still retaining a distinction from the rainy image. Extensive quantitative and qualitative experiments demonstrate that our proposed deraining network outperforms existing methods on public datasets.