SpectralGPT: Spectral Foundation Model

• DOI: 10.48550/arxiv.2311.07113
• Publication Date: 2023-01-01
• Authors: Danfeng Hong; Bing Zhang; Xuyang Li; Yuxuan Li; Chenyu Li; Jing Yao; Naoto Yokoya; Hao Li; Pedram Ghamisi; Xiuping Jia; Antonio Plaza; Paolo Gamba; Jón Atli Benedikt...; Jocelyn Chanussot
• Keywords: RGB color model

The foundation model has recently garnered significant attention due to its potential revolutionize the field of visual representation learning in a self-supervised manner. While most models are tailored effectively process RGB images for various tasks, there is noticeable gap research focused on spectral data, which offers valuable information scene understanding, especially remote sensing (RS) applications. To fill this gap, we created first time universal RS model, named SpectralGPT, purpose-built handle using novel 3D generative pretrained transformer (GPT). Compared existing models, SpectralGPT 1) accommodates input with varying sizes, resolutions, series, and regions progressive training fashion, enabling full utilization extensive big data; 2) leverages token generation spatial-spectral coupling; 3) captures spectrally sequential patterns via multi-target reconstruction; 4) trains one million images, yielding over 600 parameters. Our evaluation highlights performance improvements signifying substantial advancing data applications within geoscience across four downstream tasks: single/multi-label classification, semantic segmentation, change detection.