Cross-modal Variational Alignment of Latent Spaces

In this paper, we propose a novel cross-modal variational alignment method in order to process and relate information across different modalities. The proposed approach consists of two variational autoencoder (VAE) networks which generate and model the latent space of each modality. The first networ...

Celý popis

Uloženo v:
Podrobná bibliografie
Vydáno v:IEEE Computer Society Conference on Computer Vision and Pattern Recognition workshops s. 4127 - 4136
Hlavní autoři: Theodoridis, Thomas, Chatzis, Theocharis, Solachidis, Vassilios, Dimitropoulos, Kosmas, Daras, Petros
Médium: Konferenční příspěvek
Jazyk:angličtina
Vydáno: IEEE 01.06.2020
Témata:
Decoding
Gallium nitride
Pose estimation
Probability distribution
Task analysis
Three-dimensional displays
Training
ISSN:2160-7516
On-line přístup:Získat plný text
Tagy: Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
Popis
Shrnutí:In this paper, we propose a novel cross-modal variational alignment method in order to process and relate information across different modalities. The proposed approach consists of two variational autoencoder (VAE) networks which generate and model the latent space of each modality. The first network is a multi modal variational autoencoder that maps directly one modality to the other, while the second one is a single-modal variational autoencoder. In order to associate the two spaces, we apply variational alignment, which acts as a translation mechanism that projects the latent space of the first VAE onto the one of the single-modal VAE through an intermediate distribution. Experimental results on four well-known datasets, covering two different application domains (food image analysis and 3D hand pose estimation), show the generality of the proposed method and its superiority against a number of state-of-the-art approaches.
ISSN:2160-7516
DOI:10.1109/CVPRW50498.2020.00488