Cross-modal Variational Alignment of Latent Spaces

In this paper, we propose a novel cross-modal variational alignment method in order to process and relate information across different modalities. The proposed approach consists of two variational autoencoder (VAE) networks which generate and model the latent space of each modality. The first networ...

Full description

Saved in:
Bibliographic Details
Published in:IEEE Computer Society Conference on Computer Vision and Pattern Recognition workshops pp. 4127 - 4136
Main Authors: Theodoridis, Thomas, Chatzis, Theocharis, Solachidis, Vassilios, Dimitropoulos, Kosmas, Daras, Petros
Format: Conference Proceeding
Language:English
Published: IEEE 01.06.2020
Subjects:
Decoding
Gallium nitride
Pose estimation
Probability distribution
Task analysis
Three-dimensional displays
Training
ISSN:2160-7516
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:In this paper, we propose a novel cross-modal variational alignment method in order to process and relate information across different modalities. The proposed approach consists of two variational autoencoder (VAE) networks which generate and model the latent space of each modality. The first network is a multi modal variational autoencoder that maps directly one modality to the other, while the second one is a single-modal variational autoencoder. In order to associate the two spaces, we apply variational alignment, which acts as a translation mechanism that projects the latent space of the first VAE onto the one of the single-modal VAE through an intermediate distribution. Experimental results on four well-known datasets, covering two different application domains (food image analysis and 3D hand pose estimation), show the generality of the proposed method and its superiority against a number of state-of-the-art approaches.
ISSN:2160-7516
DOI:10.1109/CVPRW50498.2020.00488