Spatio-Temporal Encoder-Decoder Fully Convolutional Network for Video-Based Dimensional Emotion Recognition

Video-based dimensional emotion recognition aims to map human affect into the dimensional emotion space based on visual signals, which is a fundamental challenge in affective computing and human-computer interaction. In this paper, we present a novel encoder-decoder framework to tackle this problem....

Celý popis

Uložené v:
Podrobná bibliografia
Vydané v:IEEE transactions on affective computing Ročník 12; číslo 3; s. 565 - 578
Hlavní autori: Du, Zhengyin, Wu, Suowei, Huang, Di, Li, Weixin, Wang, Yunhong
Médium: Journal Article
Jazyk:English
Vydavateľské údaje: Piscataway IEEE 01.07.2021
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Predmet:
Affective computing
Artificial neural networks
Coders
Convolution
Decoding
Dimensional emotion recognition
Emotion recognition
Emotions
Encoders-Decoders
Feature extraction
spatio-temporal fully convolutional network
Task analysis
temporal hourglass CNN
temporal intermediate supervision
Videos
Visual signals
Visualization
ISSN:1949-3045, 1949-3045
On-line prístup:Získať plný text
Tagy: Pridať tag
Žiadne tagy, Buďte prvý, kto otaguje tento záznam!
Popis
Shrnutí:Video-based dimensional emotion recognition aims to map human affect into the dimensional emotion space based on visual signals, which is a fundamental challenge in affective computing and human-computer interaction. In this paper, we present a novel encoder-decoder framework to tackle this problem. It adopts a fully convolutional design with the cascaded 2D convolution based spatial encoder and 1D convolution based temporal encoder-decoder for joint spatio-temporal modeling. In particular, to address the key issue of capturing discriminative long-term dynamic dependency, our temporal model, referred to as Temporal Hourglass Convolutional Neural Network (TH-CNN), extracts contextual relationship through integrating both low-level encoded and high-level decoded clues. Temporal Intermediate Supervision (TIS) is then introduced to enhance affective representations generated by TH-CNN under a multi-resolution strategy, which guides TH-CNN to learn macroscopic long-term trend and refined short-term fluctuations progressively. Furthermore, thanks to TH-CNN and TIS, knowledge learnt from the intermediate layers also makes it possible to offer customized solutions to different applications by adjusting the decoder depth. Extensive experiments are conducted on three benchmark databases (RECOLA, SEWA and OMG) and superior results are shown compared to state-of-the-art methods, which indicates the effectiveness of the proposed approach.
Bibliografia:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:1949-3045
1949-3045
DOI:10.1109/TAFFC.2019.2940224