Unsupervised and Unregistered Hyperspectral Image Super-Resolution With Mutual Dirichlet-Net

Hyperspectral images (HSIs) provide rich spectral information that has contributed to the successful performance improvement of numerous computer vision and remote sensing tasks. However, it can only be achieved at the expense of images' spatial resolution. HSI super-resolution (HSI-SR), thus,...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on geoscience and remote sensing Vol. 60; pp. 1 - 18
Main Authors: Qu, Ying, Qi, Hairong, Kwan, Chiman, Yokoya, Naoto, Chanussot, Jocelyn
Format: Journal Article
Language:English
Published: New York IEEE 01.01.2022
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Institute of Electrical and Electronics Engineers
Subjects:
Accuracy
Bayes methods
Coders
Computer Science
Computer vision
Data mining
Dimensions
Dirichlet problem
Hyperspectral image (HSI)
Hyperspectral imaging
Image reconstruction
Image resolution
Information processing
mutual information (MI)
Optimization
Registration
Remote sensing
Representations
Resolution
Sensors
Signal and Image Processing
Spatial data
Spatial discrimination
Spatial resolution
Spectra
super-resolution (SR)
Superresolution
Tensors
unregistered
unsupervised deep learning
mutual information (MI)
Hyperspectral image (HSI)
unregistered
super-resolution (SR)
unsupervised deep learning
ISSN:0196-2892, 1558-0644
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Hyperspectral images (HSIs) provide rich spectral information that has contributed to the successful performance improvement of numerous computer vision and remote sensing tasks. However, it can only be achieved at the expense of images' spatial resolution. HSI super-resolution (HSI-SR), thus, addresses this problem by fusing low-resolution (LR) HSI with the multispectral image (MSI) carrying much higher spatial resolution (HR). Existing HSI-SR approaches require the LR HSI and HR MSI to be well registered, and the reconstruction accuracy of the HR HSI relies heavily on the registration accuracy of different modalities. In this article, we propose an unregistered and unsupervised mutual Dirichlet-Net (<inline-formula> <tex-math notation="LaTeX">u^{2} </tex-math></inline-formula>-MDN) to exploit the uncharted problem domain of HSI-SR without the requirement of multimodality registration . The success of this endeavor would largely facilitate the deployment of HSI-SR since registration requirement is difficult to satisfy in real-world sensing devices. The novelty of this work is threefold. First, to stabilize the fusion procedure of two unregistered modalities, the network is designed to extract spatial information and spectral information of two modalities with different dimensions through a shared encoder-decoder structure. Second, the mutual information (MI) is further adopted to capture the nonlinear statistical dependencies between the representations from two modalities (carrying spatial information) and their raw inputs. By maximizing the MI, spatial correlations between different modalities can be well characterized to further reduce the spectral distortion. We assume that the representations follow a similar Dirichlet distribution for their inherent sum-to-one and nonnegative properties. Third, a collaborative <inline-formula> <tex-math notation="LaTeX">l_{2,1} </tex-math></inline-formula>-norm is employed as the reconstruction error instead of the more common <inline-formula> <tex-math notation="LaTeX">l_{2} </tex-math></inline-formula>-norm to better preserve the spectral information. Extensive experimental results demonstrate the superior performance of <inline-formula> <tex-math notation="LaTeX">u^{2} </tex-math></inline-formula>-MDN as compared to the state of the art.
Bibliography:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:0196-2892
1558-0644
DOI:10.1109/TGRS.2021.3079518