STMemAE: An Instance-Level Based Spatio-Temporal Memory Autoencoder for Unsupervised Vision-Based Seizure Detection

Electroencephalogram (EEG) is most favorable in epilepsy analysis, but suffered from inconvenient recording and ease in disturbation. Contrastively, the vision-based seizure detection is more feasible in real applications for 24/7 monitoring. However, most vision-based seizure detections follow the...

Full description

Saved in:

Bibliographic Details
Published in:	IEEE transactions on emerging topics in computational intelligence Vol. 9; no. 5; pp. 3298 - 3310
Main Authors:	Hu, Dinghan, Wu, Kai, Fang, Yuan, Jiang, Tiejia, Gao, Feng, Cao, Jiuwen
Format:	Journal Article
Language:	English
Published:	Piscataway IEEE 01.10.2025 The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subjects:	Annotations Brain modeling Computer vision Convulsions & seizures Electroencephalography Epilepsy Feature extraction gradient loss Image reconstruction instance-level Machine learning Memory modules Motion perception Object recognition Prototypes Seizures STMemAE Training Unsupervised learning unsupervised seizure detection Visualization YOLO
ISSN:	2471-285X, 2471-285X
Online Access:	Get full text
Tags:	Add Tag No Tags, Be the first to tag this record!

Abstract	Electroencephalogram (EEG) is most favorable in epilepsy analysis, but suffered from inconvenient recording and ease in disturbation. Contrastively, the vision-based seizure detection is more feasible in real applications for 24/7 monitoring. However, most vision-based seizure detections follow the supervised training scheme, that is usually tedious and time-consuming in data annotations. With these regards, an effective instance-level based spatio-temporal memory autoencoder, called STMemAE, is proposed for unsupervised vision-based seizure detection in this paper. In STMemAE, YOLOv5 object detection algorithm is first applied to build an instance-level based memory AE since the frame-level based AE cannot well adapt to new scenarios. A convolutional encoder is then used to extract both the posture and motion features of the subjects. A two-stream memory net aiming to generate both spatial and temporal memory-augmented features is further developed to linearly combine the memory items. Finally, the outputs of two streams are delivered to decoder for detection. Moreover, the gradient loss is used together with the intensity loss in the training stage for better future frame generation on interictal data, which can also speedup the network training. The performance is evaluated on the video dataset recorded in Children's Hospital, Zhejiang University School of Medicine (CHZU), consisting video sequences of 15 childhood epilepsy patients. Results show that STMemAE can achieve 98.16% of Area Under Curve (AUC), and outperforms several popular deep learning models as well as many unsupervised vision-based seizure detection methods.
AbstractList	Electroencephalogram (EEG) is most favorable in epilepsy analysis, but suffered from inconvenient recording and ease in disturbation. Contrastively, the vision-based seizure detection is more feasible in real applications for 24/7 monitoring. However, most vision-based seizure detections follow the supervised training scheme, that is usually tedious and time-consuming in data annotations. With these regards, an effective instance-level based spatio-temporal memory autoencoder, called STMemAE, is proposed for unsupervised vision-based seizure detection in this paper. In STMemAE, YOLOv5 object detection algorithm is first applied to build an instance-level based memory AE since the frame-level based AE cannot well adapt to new scenarios. A convolutional encoder is then used to extract both the posture and motion features of the subjects. A two-stream memory net aiming to generate both spatial and temporal memory-augmented features is further developed to linearly combine the memory items. Finally, the outputs of two streams are delivered to decoder for detection. Moreover, the gradient loss is used together with the intensity loss in the training stage for better future frame generation on interictal data, which can also speedup the network training. The performance is evaluated on the video dataset recorded in Children's Hospital, Zhejiang University School of Medicine (CHZU), consisting video sequences of 15 childhood epilepsy patients. Results show that STMemAE can achieve 98.16% of Area Under Curve (AUC), and outperforms several popular deep learning models as well as many unsupervised vision-based seizure detection methods.
Author	Hu, Dinghan Wu, Kai Fang, Yuan Gao, Feng Jiang, Tiejia Cao, Jiuwen
Author_xml	– sequence: 1 givenname: Dinghan orcidid: 0000-0003-1493-0041 surname: Hu fullname: Hu, Dinghan email: hdh@hdu.edu.cn organization: Machine Learning and I-health International Cooperation Base of Zhejiang Province, Hangzhou Dianzi University, Hangzhou, China – sequence: 2 givenname: Kai surname: Wu fullname: Wu, Kai email: 396395584@qq.com organization: Machine Learning and I-health International Cooperation Base of Zhejiang Province, Hangzhou Dianzi University, Hangzhou, China – sequence: 3 givenname: Yuan orcidid: 0000-0002-8070-2680 surname: Fang fullname: Fang, Yuan email: fangyuan2017@zjnu.edu.cn organization: Machine Learning and I-health International Cooperation Base of Zhejiang Province, Hangzhou Dianzi University, Hangzhou, China – sequence: 4 givenname: Tiejia orcidid: 0000-0003-3688-8366 surname: Jiang fullname: Jiang, Tiejia email: jiangyouze@zju.edu.cn organization: Department of Neurology, Children's Hospital, Zhejiang University, Hangzhou, China – sequence: 5 givenname: Feng orcidid: 0000-0003-4907-7212 surname: Gao fullname: Gao, Feng email: epilepsy@zju.edu.cn organization: Department of Neurology, Children's Hospital, Zhejiang University, Hangzhou, China – sequence: 6 givenname: Jiuwen orcidid: 0000-0002-6480-5794 surname: Cao fullname: Cao, Jiuwen email: jwcao@hdu.edu.cn organization: Machine Learning and I-health International Cooperation Base of Zhejiang Province, Hangzhou Dianzi University, Hangzhou, China
BookMark	eNp9kEFPwjAYhhuDiYj8AeOhiedh-3VlqzdUVBKMB8B4W8r2LSmBdrYbCf56h3AgHjz1O7zP27zPJelYZ5GQa84GnDN1Nx_PHycDYBAPhAQAlp6RLsQJjyCVn52T-4L0Q1gxxkBJLmTcJWE2f8PNaHxPR5ZObKi1zTGa4hbX9EEHLOis0rVx0Rw3lfN6Tdu48zs6amqHNncFelo6Txc2NBX6rdkzHyYYZ6NjAZrvxiN9whrztspekfNSrwP2j2-PLJ7bCa_R9P1l8jiaRjmoYR0lhVgqHSepKAVioRKVLFMsIAatOegSeVHoWOaJRIkJ40OWqnKpMB4WoIXIRY_cHnor774aDHW2co237ZeZACkUE1JAm0oPqdy7EDyWWW7q_WRbe23WGWfZ3nL2aznbW86OllsU_qCVNxvtd_9DNwfIIOIJkMJQtlt_AJ1fi6I
CODEN	ITETCU
CitedBy_id	crossref_primary_10_1080_01616412_2025_2555516
Cites_doi	10.1109/TNSRE.2021.3107142 10.1109/ISSPIT.2018.8642670 10.1109/TCDS.2021.3064228 10.1109/ICASSP40776.2020.9054649 10.1117/12.2243622 10.1109/TCDS.2019.2936441 10.1109/jbhi.2015.2462079 10.1109/ICMIC.2016.7804256 10.1109/tim.2022.3220287 10.1109/TNSRE.2022.3229066 10.1016/S0140-6736(18)32596-0 10.1007/s10489-022-03613-1 10.1109/TPAMI.2016.2577031 10.1109/TCSVT.2020.3027843 10.1109/TCDS.2022.3175636 10.1109/ICCV.2017.74 10.48550/arXiv.1804.02767 10.1109/TETCI.2022.3171855 10.5698/1535-7511.17.3.180 10.1109/EMBC.2012.6346955 10.1109/TITB.2012.2186586 10.1109/TCSVT.2019.2962229 10.1109/TCSII.2022.3188162 10.1109/ICASSP43922.2022.9746325 10.1109/IEMENTECH.2017.8076983 10.1016/j.isatra.2022.07.014 10.1016/j.neunet.2022.03.014 10.1109/JBHI.2019.2895855 10.1109/ISBI.2018.8363746 10.1109/cvpr.2018.00684 10.1016/j.cmpb.2020.105472 10.1109/CVPR.2018.00745 10.1109/WoWMoM.2011.5986193 10.1111/j.0013-9580.2005.66104.x 10.1109/ICC.2012.6364396 10.1109/TCSVT.2022.3195727 10.1109/CVPR42600.2020.01438 10.1109/TCSVT.2022.3221622 10.1109/TIP.2021.3096089 10.1007/s13755-020-00129-1 10.1016/j.neunet.2019.11.023 10.1109/TNSRE.2022.3223056 10.1109/JBHI.2021.3049649 10.1109/ICCV.2019.00718 10.1109/CVPR46437.2021.01517 10.1109/TCSVT.2020.3039798 10.1109/ICCV.2019.00179 10.1016/j.cmpb.2021.106604 10.1002/sam.11161
ContentType	Journal Article
Copyright	Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2025
Copyright_xml	– notice: Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2025
DBID	97E RIA RIE AAYXX CITATION 7SP 8FD L7M
DOI	10.1109/TETCI.2024.3522208
DatabaseName	IEEE All-Society Periodicals Package (ASPP) 2005–Present IEEE All-Society Periodicals Package (ASPP) 1998–Present IEEE/IET Electronic Library (IEL) (UW System Shared) CrossRef Electronics & Communications Abstracts Technology Research Database Advanced Technologies Database with Aerospace
DatabaseTitle	CrossRef Technology Research Database Advanced Technologies Database with Aerospace Electronics & Communications Abstracts
DatabaseTitleList	Technology Research Database
Database_xml	– sequence: 1 dbid: RIE name: IEEE/IET Electronic Library (IEL) (UW System Shared) url: https://ieeexplore.ieee.org/ sourceTypes: Publisher
DeliveryMethod	fulltext_linktorsrc
EISSN	2471-285X
EndPage	3310
ExternalDocumentID	10_1109_TETCI_2024_3522208 10826578
Genre	orig-research
GrantInformation_xml	– fundername: National Natural Science Foundation of China grantid: U1909209 funderid: 10.13039/501100001809 – fundername: Natural Science Key Foundation of Zhejiang Province grantid: LZ24F030010 – fundername: National Key Research and Development Program of China grantid: 2021YFE0100100
GroupedDBID	0R~ 97E AAJGR AASAJ AAWTH ABAZT ABJNI ABQJQ ABVLG ACGFS AGQYO AHBIQ AKJIK AKQYR ALMA_UNASSIGNED_HOLDINGS ATWAV BEFXN BFFAM BGNUA BKEBE BPEOZ EBS EJD IFIPE JAVBF OCL RIA RIE AAYXX CITATION 7SP 8FD L7M
ID	FETCH-LOGICAL-c296t-7d3b9a4783f3eed9797b8ed242aa12afe1dda45c75e5e7016089fb9e46d2a33c3
IEDL.DBID	RIE
ISICitedReferencesCount	1
ISICitedReferencesURI	http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=001395289200001&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
ISSN	2471-285X
IngestDate	Thu Nov 27 15:42:26 EST 2025 Sat Nov 29 07:25:36 EST 2025 Tue Nov 18 22:18:52 EST 2025 Wed Oct 01 07:05:09 EDT 2025
IsPeerReviewed	true
IsScholarly	true
Issue	5
Language	English
License	https://ieeexplore.ieee.org/Xplorehelp/downloads/license-information/IEEE.html https://doi.org/10.15223/policy-029 https://doi.org/10.15223/policy-037
LinkModel	DirectLink
MergedId	FETCHMERGED-LOGICAL-c296t-7d3b9a4783f3eed9797b8ed242aa12afe1dda45c75e5e7016089fb9e46d2a33c3
Notes	ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 14
ORCID	0000-0002-6480-5794 0000-0003-1493-0041 0000-0003-4907-7212 0000-0003-3688-8366 0000-0002-8070-2680
PQID	3253903532
PQPubID	4437216
PageCount	13
ParticipantIDs	ieee_primary_10826578 proquest_journals_3253903532 crossref_citationtrail_10_1109_TETCI_2024_3522208 crossref_primary_10_1109_TETCI_2024_3522208
PublicationCentury	2000
PublicationDate	2025-10-01
PublicationDateYYYYMMDD	2025-10-01
PublicationDate_xml	– month: 10 year: 2025 text: 2025-10-01 day: 01
PublicationDecade	2020
PublicationPlace	Piscataway
PublicationPlace_xml	– name: Piscataway
PublicationTitle	IEEE transactions on emerging topics in computational intelligence
PublicationTitleAbbrev	TETCI
PublicationYear	2025
Publisher	IEEE The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher_xml	– name: IEEE – name: The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
References	ref13 ref15 ref14 ref52 ref11 ref10 ref17 ref16 ref19 ref18 Hilal (ref24) 2021; 193 ref51 ref50 ref45 ref48 ref47 ref42 ref41 ref44 ref43 ref49 ref8 ref7 ref9 ref4 ref3 ref6 ref5 ref40 ref35 ref34 ref37 ref36 ref31 ref30 ref33 ref32 Raghu (ref12) 2020; 124 ref2 ref1 ref39 ref38 Jocher (ref46) 2020 ref23 ref26 ref25 ref20 ref22 ref21 ref28 ref27 ref29
References_xml	– ident: ref8 doi: 10.1109/TNSRE.2021.3107142 – volume: 124 start-page: 202 volume-title: Neural Netw. year: 2020 ident: ref12 article-title: EEG based multi-class seizure type classification using convolutional neural network and transfer learning – ident: ref35 doi: 10.1109/ISSPIT.2018.8642670 – ident: ref3 doi: 10.1109/TCDS.2021.3064228 – ident: ref20 doi: 10.1109/ICASSP40776.2020.9054649 – ident: ref40 doi: 10.1117/12.2243622 – ident: ref6 doi: 10.1109/TCDS.2019.2936441 – ident: ref14 doi: 10.1109/jbhi.2015.2462079 – ident: ref39 doi: 10.1109/ICMIC.2016.7804256 – ident: ref4 doi: 10.1109/tim.2022.3220287 – volume: 193 year: 2021 ident: ref24 article-title: A review of anomaly detection techniques and applications in financial fraud publication-title: Expert Syst. Appl. – ident: ref9 doi: 10.1109/TNSRE.2022.3229066 – ident: ref1 doi: 10.1016/S0140-6736(18)32596-0 – ident: ref42 doi: 10.1007/s10489-022-03613-1 – ident: ref51 doi: 10.1109/TPAMI.2016.2577031 – ident: ref31 doi: 10.1109/TCSVT.2020.3027843 – ident: ref5 doi: 10.1109/TCDS.2022.3175636 – ident: ref47 doi: 10.1109/ICCV.2017.74 – ident: ref52 doi: 10.48550/arXiv.1804.02767 – ident: ref16 doi: 10.1109/TETCI.2022.3171855 – ident: ref17 doi: 10.5698/1535-7511.17.3.180 – ident: ref18 doi: 10.1109/EMBC.2012.6346955 – ident: ref34 doi: 10.1109/TITB.2012.2186586 – ident: ref44 doi: 10.1109/TCSVT.2019.2962229 – ident: ref15 doi: 10.1109/TCSII.2022.3188162 – ident: ref36 doi: 10.1109/ICASSP43922.2022.9746325 – ident: ref38 doi: 10.1109/IEMENTECH.2017.8076983 – ident: ref41 doi: 10.1016/j.isatra.2022.07.014 – ident: ref10 doi: 10.1016/j.neunet.2022.03.014 – ident: ref22 doi: 10.1109/JBHI.2019.2895855 – ident: ref19 doi: 10.1109/ISBI.2018.8363746 – ident: ref23 doi: 10.1109/cvpr.2018.00684 – ident: ref25 doi: 10.1016/j.cmpb.2020.105472 – ident: ref49 doi: 10.1109/CVPR.2018.00745 – ident: ref33 doi: 10.1109/WoWMoM.2011.5986193 – ident: ref2 doi: 10.1111/j.0013-9580.2005.66104.x – ident: ref37 doi: 10.1109/ICC.2012.6364396 – ident: ref32 doi: 10.1109/TCSVT.2022.3195727 – ident: ref29 doi: 10.1109/CVPR42600.2020.01438 – ident: ref45 doi: 10.1109/TCSVT.2022.3221622 – ident: ref50 doi: 10.1109/TIP.2021.3096089 – ident: ref13 doi: 10.1007/s13755-020-00129-1 – ident: ref11 doi: 10.1016/j.neunet.2019.11.023 – year: 2020 ident: ref46 article-title: YOLOv5: V3.1 - Bug fixes and performance improvements – ident: ref7 doi: 10.1109/TNSRE.2022.3223056 – ident: ref21 doi: 10.1109/JBHI.2021.3049649 – ident: ref48 doi: 10.1109/ICCV.2019.00718 – ident: ref30 doi: 10.1109/CVPR46437.2021.01517 – ident: ref43 doi: 10.1109/TCSVT.2020.3039798 – ident: ref28 doi: 10.1109/ICCV.2019.00179 – ident: ref26 doi: 10.1016/j.cmpb.2021.106604 – ident: ref27 doi: 10.1002/sam.11161
SSID	ssj0002951354
Score	2.324758
Snippet	Electroencephalogram (EEG) is most favorable in epilepsy analysis, but suffered from inconvenient recording and ease in disturbation. Contrastively, the...
SourceID	proquest crossref ieee
SourceType	Aggregation Database Enrichment Source Index Database Publisher
StartPage	3298
SubjectTerms	Annotations Brain modeling Computer vision Convulsions & seizures Electroencephalography Epilepsy Feature extraction gradient loss Image reconstruction instance-level Machine learning Memory modules Motion perception Object recognition Prototypes Seizures STMemAE Training Unsupervised learning unsupervised seizure detection Visualization YOLO
Title	STMemAE: An Instance-Level Based Spatio-Temporal Memory Autoencoder for Unsupervised Vision-Based Seizure Detection
URI	https://ieeexplore.ieee.org/document/10826578 https://www.proquest.com/docview/3253903532
Volume	9
WOSCitedRecordID	wos001395289200001&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
hasFullText	1
inHoldings	1
isFullTextHit
isPrint
journalDatabaseRights	– providerCode: PRVIEE databaseName: IEEE/IET Electronic Library (IEL) (UW System Shared) customDbUrl: eissn: 2471-285X dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0002951354 issn: 2471-285X databaseCode: RIE dateStart: 20170101 isFulltext: true titleUrlDefault: https://ieeexplore.ieee.org/ providerName: IEEE
link	http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1LS8QwEA66ePDiAxXXFzl4k2ibpE3jrb5wQUWwireSJrMgaFf6EPTXm6TdZUEUvPWQKW2_SeZL-s0MQofaxNaPdEIUM4ZwLjRJQiiIJR_GhmeWKF94_ulG3N0lz8_yvk9W97kwAODFZ3DsLv2_fDPRrTsqszPckmHrYotoUYi4S9aaHahQyxVYxKeJMYE8yS6z85HdAlJ-7GgGdS0k54KP76byYwn2ceVq9Z9PtIZWegKJ0w7xdbQA5QaqH7JbeEsvT3Fa4pFnfBrIjRME4TMbpwx-8MppknWVqF7xrVPYfuK0bSaulKWBClv6ih_Lun13y4ezefJ556S_Abx8tRXgC2i8eqvcRI9X9tWvSd9OgWgq44YIwwqpuEjYmNnIKIUURQLGxmilQqrGEBqjeKRFBBEIV3kukeNCAo8NVYxptoUG5aSEbYRZEIAuqEVVaa6VkRCbCLxmNBxHARuicPqdc93XGnctL15zv-cIZO6xyR02eY_NEB3NbN67Sht_jt50aMyN7IAYor0pnnk_G-uc0YjJgEWM7vxitouWqWvs61V6e2jQVC3soyX90bzU1YF3tG_vs9I9
linkProvider	IEEE
linkToHtml	http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1bS8MwFA7eQF-8oOK85sE3yWxzWRvf5g2H2xDsxt5KmpyBoJ1sraC_3iTtRBAF3_qQU9p-Jzlf0u-cg9CpNi3rRzomihlDOI80iUPIiCUfxoZnFitfeH7Yjfr9eDSSD3Wyus-FAQAvPoOmu_T_8s1El-6ozM5wS4atiy2iZcE5Dap0ra8jFWrZAhN8nhoTyPPkJrnq2E0g5U1HNKhrIvkt_Ph-Kj8WYR9Zbjf--UybaL2mkLhdYb6FFiDfRrPHpAcv7ZsL3M5xx3M-DaTrJEH40kYqgx-9dpokVS2qZ9xzGtt33C6LiStmaWCKLYHFg3xWvroFxNkMfeY5qW8ATx_lFPA1FF6_le-gwa199TtSN1QgmspWQSLDMql4FLMxs7FRRjLKYjA2SisVUjWG0BjFhY4ECIhc7blYjjMJvGWoYkyzXbSUT3LYQ5gFAeiMWlyV5loZCS0jwKtGw7EIWAOF8--c6rrauGt68Zz6XUcgU49N6rBJa2wa6OzL5rWqtfHn6B2HxreRFRANdDjHM63n4yxlVDAZMMHo_i9mJ2j1Lul1026nf3-A1qhr8-s1e4doqZiWcIRW9FvxNJsee6f7BAW41YQ
openUrl	ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=STMemAE%3A+An+Instance-Level+Based+Spatio-Temporal+Memory+Autoencoder+for+Unsupervised+Vision-Based+Seizure+Detection&rft.jtitle=IEEE+transactions+on+emerging+topics+in+computational+intelligence&rft.au=Hu%2C+Dinghan&rft.au=Wu%2C+Kai&rft.au=Fang%2C+Yuan&rft.au=Jiang%2C+Tiejia&rft.date=2025-10-01&rft.issn=2471-285X&rft.eissn=2471-285X&rft.volume=9&rft.issue=5&rft.spage=3298&rft.epage=3310&rft_id=info:doi/10.1109%2FTETCI.2024.3522208&rft.externalDBID=n%2Fa&rft.externalDocID=10_1109_TETCI_2024_3522208
thumbnail_l	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=2471-285X&client=summon
thumbnail_m	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=2471-285X&client=summon
thumbnail_s	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=2471-285X&client=summon