Unsupervised Multispectral and Hyperspectral Image Fusion with Deep Spatial and Spectral Priors

Zhe Liu, Yinqiang Zheng, Xian-Hua Han; Proceedings of the Asian Conference on Computer Vision (ACCV) Workshops, 2020

Hyperspectral (HS) imaging is a promising imaging modality, which can simultaneously acquire various bands of images of the same scene and capture detailed spectral distribution helping for numerous applications. However, existing HS imaging sensor can only obtain images with low spatial resolution. Thus fusing a low resolution hyperspectral (LR-HS) image with a high resolution (HR) RGB (or multispectral) image into a HR-HS image has received much attention. Conventional fusion methods usually employ various hand-crafted priors to regularize the mathematical model formulating the relation between the observations and the HR-HS image, and conduct optimization for pursuing the optimal solution. However, the politic prior would be various for different scenes and is difficult to hammer out for a specific scene. Recently, deep learning-based methods have been widely explored for HS image resolution enhancement, and impressive performance has been validated. As it is known that deep learning-based methods essentially require large-scale training samples, which are hard to obtain due to the limitation of the existing HS cameras, for constructing the model with good generalization. Motivated by the deep image prior that network architecture itself sufficiently captures a great deal of low-level image statistics with arbitrary learning strategy, we investigate the deep learned image prior consisting both spatial structure and spectral attribute instead of hand-crafted priors for unsupervised multispectral (RGB) and HS image fusion, and propose a novel deep spatial and spectral prior learning framework for exploring the underlying structure of the latent HR-HS image with the observed HR-RGB and LR-HS images only. The proposed deep prior learning method has no requirement to prepare massive triplets of the HR-RGB, LR-HS and HR-HS images for network training. We validate the proposed method on two benchmark HS image datasets, and experimental results show that our method is comparable or outperforms the state-of-the-art HS image super-resolution approaches.