Cross-modal Variational Alignment of Latent Spaces

In this paper, we propose a novel cross-modal variational alignment method in order to process and relate information across different modalities. The proposed approach consists of two variational autoencoder (VAE) networks which generate and model the latent space of each modality. The first networ...

Celý popis

Uložené v:
Podrobná bibliografia
Vydané v:IEEE Computer Society Conference on Computer Vision and Pattern Recognition workshops s. 4127 - 4136
Hlavní autori: Theodoridis, Thomas, Chatzis, Theocharis, Solachidis, Vassilios, Dimitropoulos, Kosmas, Daras, Petros
Médium: Konferenčný príspevok..
Jazyk:English
Vydavateľské údaje: IEEE 01.06.2020
Predmet:
Decoding
Gallium nitride
Pose estimation
Probability distribution
Task analysis
Three-dimensional displays
Training
ISSN:2160-7516
On-line prístup:Získať plný text
Tagy: Pridať tag
Žiadne tagy, Buďte prvý, kto otaguje tento záznam!
Popis
Shrnutí:In this paper, we propose a novel cross-modal variational alignment method in order to process and relate information across different modalities. The proposed approach consists of two variational autoencoder (VAE) networks which generate and model the latent space of each modality. The first network is a multi modal variational autoencoder that maps directly one modality to the other, while the second one is a single-modal variational autoencoder. In order to associate the two spaces, we apply variational alignment, which acts as a translation mechanism that projects the latent space of the first VAE onto the one of the single-modal VAE through an intermediate distribution. Experimental results on four well-known datasets, covering two different application domains (food image analysis and 3D hand pose estimation), show the generality of the proposed method and its superiority against a number of state-of-the-art approaches.
ISSN:2160-7516
DOI:10.1109/CVPRW50498.2020.00488