Cross-modal Variational Alignment of Latent Spaces

In this paper, we propose a novel cross-modal variational alignment method in order to process and relate information across different modalities. The proposed approach consists of two variational autoencoder (VAE) networks which generate and model the latent space of each modality. The first networ...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:IEEE Computer Society Conference on Computer Vision and Pattern Recognition workshops S. 4127 - 4136
Hauptverfasser: Theodoridis, Thomas, Chatzis, Theocharis, Solachidis, Vassilios, Dimitropoulos, Kosmas, Daras, Petros
Format: Tagungsbericht
Sprache:Englisch
Veröffentlicht: IEEE 01.06.2020
Schlagworte:
Decoding
Gallium nitride
Pose estimation
Probability distribution
Task analysis
Three-dimensional displays
Training
ISSN:2160-7516
Online-Zugang:Volltext
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Beschreibung
Zusammenfassung:In this paper, we propose a novel cross-modal variational alignment method in order to process and relate information across different modalities. The proposed approach consists of two variational autoencoder (VAE) networks which generate and model the latent space of each modality. The first network is a multi modal variational autoencoder that maps directly one modality to the other, while the second one is a single-modal variational autoencoder. In order to associate the two spaces, we apply variational alignment, which acts as a translation mechanism that projects the latent space of the first VAE onto the one of the single-modal VAE through an intermediate distribution. Experimental results on four well-known datasets, covering two different application domains (food image analysis and 3D hand pose estimation), show the generality of the proposed method and its superiority against a number of state-of-the-art approaches.
ISSN:2160-7516
DOI:10.1109/CVPRW50498.2020.00488