Designing CNNs for Multimodal Image Super-Resolution via the Method of Multipliers

Multimodal alias, guided, image super-resolution (SR) refers to the reconstruction of a high-resolution (HR) version of a low-resolution (LR) image with the aid of an HR image from another image modality. Common approaches for the SR problem include analytical methods which are computationally expen...

Full description

Saved in:
Bibliographic Details
Published in:2020 28th European Signal Processing Conference (EUSIPCO) pp. 780 - 783
Main Authors: Marivani, Iman, Tsiligianni, Evaggelia, Cornelis, Bruno, Deligiannis, Nikos
Format: Conference Proceeding
Language:English
Published: Eurasip 24.01.2021
Subjects:
Convolution
Convolutional codes
Deep learning
deep unfolding
Image coding
Image reconstruction
Method of multipliers
multimodal CNN
multimodal image super-resolution
Signal processing algorithms
Signal resolution
ISSN:2076-1465
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Multimodal alias, guided, image super-resolution (SR) refers to the reconstruction of a high-resolution (HR) version of a low-resolution (LR) image with the aid of an HR image from another image modality. Common approaches for the SR problem include analytical methods which are computationally expensive. Deep learning methods are capable of learning a nonlinear mapping between LR and HR images from data, delivering high reconstruction accuracy at a low-computational cost during inference; however, these methods do not incorporate any prior knowledge about the problem, with the neural network model behaving like a black box. In this paper, we formulate multimodal image SR as a coupled convolutional sparse coding problem. To solve the corresponding minimization problem, we adopt the Method of Multipliers (MM). We then design a convolutional neural network (CNN) that unfolds the obtained MM algorithm. The proposed CNN accepts as input the LR image from the main modality and the HR image from the guidance modality to reconstruct the desired HR image. Unlike existing deep learning methods, our CNN provides an efficient and structured way to fuse information at different stages of the network and achieves high reconstruction accuracy. We evaluate the performance of the proposed model for the super-resolution of multi-spectral images guided by their high resolution RGB counterparts.
ISSN:2076-1465
DOI:10.23919/Eusipco47968.2020.9287361