Cross-modal Variational Alignment of Latent Spaces

Abstract:


In this paper, we propose a novel cross-modal variational alignment method in order to process and relate information across different modalities. The proposed approach consists of two variational autoencoder (VAE) networks which generate and model the latent space of each modality. The first network is a multi-modal variational autoencoder that maps directly one modality to the other, while the second one is a single-modal variational autoencoder. In order to associate the two spaces, we apply variational alignment, which acts as a translation mechanism that projects the latent space of the first VAE onto the one of the single-modal VAE through an intermediate distribution. Experimental results on four well-known datasets, covering two different application domains (food image analysis and 3D hand pose estimation), show the generality of the proposed method and its superiority against a number of state-of-the-art approaches.


  • T. Theodoridis, T. Chatzis, V. Solachidis, K. Dimitropoulos, P. Daras, "Cross-modal Variational Alignment of Latent Spaces", In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Washington, United States, June, 2020.

  • Visual Computing Lab

    The focus of the Visual Computing Laboratory is to develop new algorithms and architectures for applications in the areas of 3D processing, image/video processing, computer vision, pattern recognition, bioinformatics and medical imaging.

    Contact Information

    Dr. Petros Daras, Research Director
    6th km Charilaou – Thermi Rd, 57001, Thessaloniki, Greece
    P.O.Box: 60361
    Tel.: +30 2310 464160 (ext. 156)
    Fax: +30 2310 464164
    Email: daras(at)iti(dot)gr