Canonical Correlation Analysis for Data Fusion in Multimodal Emotion Recognition

Multimodal emotion recognition systems aim at classifying emotion data, usually from different natures, into discrete affective categories. These systems fuse different modalities as each modality classifies the data from its own viewpoint and can compensate the limitations of others when combining...

Full description

Saved in:
Bibliographic Details
Published in:2018 9th International Symposium on Telecommunications (IST) pp. 676 - 681
Main Author: Nemati, Shahla
Format: Conference Proceeding
Language:English
Published: IEEE 01.12.2018
Subjects:
CCA
Correlation
Data Fusion
Data integration
Emotion recognition
Feature extraction
Multimodal
Physiology
Synchronization
Visualization
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Multimodal emotion recognition systems aim at classifying emotion data, usually from different natures, into discrete affective categories. These systems fuse different modalities as each modality classifies the data from its own viewpoint and can compensate the limitations of others when combining with them. Existing approaches for multimodal data fusion either use feature-level or decision-level fusion. The former needs different modalities to be synchronized while the latter has not this limitation. In the current study, audio, visual, and users' comments are used as modalities for video emotion recognition. Although the first two modalities are synchronized, users' comments are not synchronized with them and this makes the use of pure feature-level data fusion impossible. In order to exploit the benefits of the feature-level approach, a hybrid method is proposed in the current study which first applies feature-level canonical correlation analysis (CCA) to audio and visual modalities and then combines the outputs with users' comments using a decision-level fusion. The results of applying this proposed method to DEAP dataset shows that using feature-level CCA not only outperforms the baseline feature-level method but also achieves a higher performance than the decision-level fusion.
DOI:10.1109/ISTEL.2018.8661140