Social-MAE: A Transformer-Based Multimodal Autoencoder for Face and Voice

Human social behaviors are inherently multi-modal necessitating the development of powerful audiovisual models for their perception. In this paper, we present Social-MAE, our pre-trained audiovisual Masked Autoencoder based on an extended version of Contrastive Audio-Visual Masked Auto-Encoder (CAV-...

Full description

Saved in:
Bibliographic Details
Published in:IEEE International Conference and Workshops on Automatic Face and Gesture Recognition : FG pp. 1 - 5
Main Authors: Bohy, Hugo, Tran, Minh, El Haddad, Kevin, Dutoit, Thierry, Soleymani, Mohammad
Format: Conference Proceeding
Language:English
Published: IEEE 27.05.2024
Subjects:
Adaptation models
Codes
Emotion recognition
Estimation
Face recognition
Gesture recognition
Transformers
ISSN:2770-8330
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Human social behaviors are inherently multi-modal necessitating the development of powerful audiovisual models for their perception. In this paper, we present Social-MAE, our pre-trained audiovisual Masked Autoencoder based on an extended version of Contrastive Audio-Visual Masked Auto-Encoder (CAV-MAE), which is pre-trained on audiovisual social data. Specifically, we modify CAV-MAE to receive a larger number of frames as input and pre-train it on a large dataset of human social interaction (VoxCeleb2) in a self-supervised manner. We demonstrate the effectiveness of this model by fine-tuning and evaluating the model on different social and affective downstream tasks, namely, emotion recognition, laughter detection and apparent personality estimation. The model achieves state-of-the-art results on multimodal emotion recognition and laughter recognition and competitive results for apparent personality estimation, demonstrating the effectiveness of in-domain self-supervised pre-training. Code and model weight are available here https://github.com/HuBohy/SocialMAE.
ISSN:2770-8330
DOI:10.1109/FG59268.2024.10581940