Enhanced Residue Prediction for Lossless Coding of Multimodal Image Pairs Based on Image-to-Image Translation - Nicolau, Daniel S.; Parracho, Joao O.; Thomaz, Lucas A.; Tavora, Luis M. N.; Faria, Sergio M. M. (European Workshop on Visual Information Processing, 2023)
Multimodal medical imaging combine data obtained from multiple techniques simultaneously, yielding more detailed information about the content, which is a clear advantage over independent acquisition techniques. As these images are acquired using different imaging modalities and sometimes even in different dimensions, they commonly require a geometrical registration process. However, when they are encoded using standard image codecs the prediction methods do not exploit the redundancies related to the multimodal acquisition. In this paper, a novel lossless multimodal prediction module is introduced. The proposed method employs a deep learning-based approach with Image-to-Image translation for the purpose of joint coding of Positron Emission Tomography (PET) and Computed Tomography (CT) image pairs. Prior to the coding stage, a Generative Adversarial Network (GAN) is used for multimodal image translation. Then, a weighted estimated image is utilised as the I-frame, while the weighted sum of the original and synthesised image from the same modality serves as the P-frame for inter prediction. By employing weighted frames, the predictive frame approximates the reference frame more accurately, enhancing the overall performance of the prediction process. The experimental results, on a publicly available PET-CT dataset, demonstrate that the proposed prediction scheme outperformed the previously proposed method, and attains coding gains up to 13.20% when compared with the single modality intra coding of the Versatile Video Coding (VVC) lossless standard.