I2I-Mamba: Multi-modal medical image synthesis via selective state space modeling

• DOI: 10.48550/arxiv.2405.14022
• Publication Date: 2024-01-01
• Authors: Atli, Omer F.; Kabas, Bilal; Arslan, Fuat; Demirtas, Arda C.; Yurt, Mahmut; Dalmaz, Onat; Çukur, Tolga
• Keywords: FOS: Computer and information sciences; Computer Vision and Pattern Recognition (cs.CV); Image and Video Processing (eess.IV); Computer Science - Computer Vision and Pattern Recognition; FOS: Electrical engineering, electronic engineering, information engineering; Electrical Engineering and Systems Science - Image and Video Processing

In recent years, deep learning models comprising transformer components have pushed the performance envelope in medical image synthesis tasks. Contrary to convolutional neural networks (CNNs) that use static, local filters, transformers use self-attention mechanisms to permit adaptive, non-local filtering to sensitively capture long-range context. However, this sensitivity comes at the expense of substantial model complexity, which can compromise learning efficacy particularly on relatively modest-sized imaging datasets. Here, we propose a novel adversarial model for multi-modal medical image synthesis, I2I-Mamba, that leverages selective state space modeling (SSM) to efficiently capture long-range context while maintaining local precision. To do this, I2I-Mamba injects channel-mixed Mamba (cmMamba) blocks in the bottleneck of a convolutional backbone. In cmMamba blocks, SSM layers are used to learn context across the spatial dimension and channel-mixing layers are used to learn context across the channel dimension of feature maps. Comprehensive demonstrations are reported for imputing missing images in multi-contrast MRI and MRI-CT protocols. Our results indicate that I2I-Mamba offers superior performance against state-of-the-art CNN- and transformer-based methods in synthesizing target-modality images.<br/>10 pages, 6 figures<br/>