Detection of Important Scenes in Baseball Videos via a Time-Lag-Aware Multimodal Variational Autoencoder

A new method for the detection of important scenes in baseball videos via a time-lag-aware multimodal variational autoencoder (Tl-MVAE) is presented in this paper. Tl-MVAE estimates latent features calculated from tweet, video, and audio features extracted from tweets and videos. Then, important sce...

Full description

Saved in:
Bibliographic Details
Published in:Sensors (Basel, Switzerland) Vol. 21; no. 6; p. 2045
Main Authors: Hirasawa, Kaito, Maeda, Keisuke, Ogawa, Takahiro, Haseyama, Miki
Format: Journal Article
Language:English
Published: Switzerland MDPI AG 14.03.2021
MDPI
Subjects:
Baseball
detection of important scenes
Influence
Methods
multimodal variational autoencoder
Neural networks
Sensors
Social networks
sports video
time-lags
Twitter
Twitter
time-lags
multimodal variational autoencoder
detection of important scenes
sports video
ISSN:1424-8220, 1424-8220
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:A new method for the detection of important scenes in baseball videos via a time-lag-aware multimodal variational autoencoder (Tl-MVAE) is presented in this paper. Tl-MVAE estimates latent features calculated from tweet, video, and audio features extracted from tweets and videos. Then, important scenes are detected by estimating the probability of the scene being important from estimated latent features. It should be noted that there exist time-lags between tweets posted by users and videos. To consider the time-lags between tweet features and other features calculated from corresponding multiple previous events, the feature transformation based on feature correlation considering such time-lags is newly introduced to the encoder in MVAE in the proposed method. This is the biggest contribution of the Tl-MVAE. Experimental results obtained from actual baseball videos and their corresponding tweets show the effectiveness of the proposed method.
Bibliography:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
content type line 23
This paper is an extended version of our paper published in: Hirasawa, K.; Maeda, K.; Ogawa, T.; Haseyama, M. Important Scene Detection Based on Anomaly Detection using Long Short-Term Memory for Baseball Highlight Generation. In the Proceedings of the IEEE International Conference on Consumer Electronics—Taiwan (IEEE 2020 ICCE-TW), Taoyuan, Taiwan, 28–30 September 2020.
ISSN:1424-8220
1424-8220
DOI:10.3390/s21062045