CSCAD: Correlation Structure-Based Collective Anomaly Detection in Complex System

Detecting anomalies in large complex systems is a critical and challenging task. The difficulties arise from several aspects. First, collecting ground truth labels or prior knowledge for anomalies is hard in real-world systems, which often lead to limited or no anomaly labels in the dataset. Second,...

Ausführliche Beschreibung

Gespeichert in:

Bibliographische Detailangaben
Veröffentlicht in:	IEEE transactions on knowledge and data engineering Jg. 35; H. 5; S. 4634 - 4645
Hauptverfasser:	Qin, Huiling, Zhan, Xianyuan, Zheng, Yu
Format:	Journal Article
Sprache:	Englisch
Veröffentlicht:	New York IEEE 01.05.2023 The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Schlagworte:	Algorithms Anomalies Anomaly detection complex system Complex systems Correlation correlation mining Data models Datasets Feature extraction Labels Loss measurement Reconstruction Sensors unsupervised learning urban computing variational autoencoder
ISSN:	1041-4347, 1558-2191
Online-Zugang:	Volltext
Tags:	Tag hinzufügen Keine Tags, Fügen Sie den ersten Tag hinzu!

Abstract	Detecting anomalies in large complex systems is a critical and challenging task. The difficulties arise from several aspects. First, collecting ground truth labels or prior knowledge for anomalies is hard in real-world systems, which often lead to limited or no anomaly labels in the dataset. Second, anomalies in large systems usually occur in a collective manner due to the underlying dependency structure among devices or sensors. Lastly, real-time anomaly detection for high-dimensional data requires efficient algorithms that are capable of handling different types of data (i.e. continuous and discrete). We propose a correlation structure-based collective anomaly detection (CSCAD) model for high-dimensional anomaly detection problem in large systems, which is also generalizable to semi-supervised or supervised settings. Our framework utilize graph convolutional network combining a variational autoencoder to jointly exploit the feature space correlation and reconstruction deficiency of samples to perform anomaly detection. We propose an extended mutual information (EMI) metric to mine the internal correlation structure among different data features, which enhances the data reconstruction capability of CSCAD. The reconstruction loss and latent standard deviation vector of a sample obtained from reconstruction network can be perceived as two natural anomalous degree measures. An anomaly discriminating network can then be trained using low anomalous degree samples as positive samples, and high anomalous degree samples as negative samples. Experimental results on five public datasets demonstrate that our approach consistently outperforms all the competing baselines.
AbstractList	Detecting anomalies in large complex systems is a critical and challenging task. The difficulties arise from several aspects. First, collecting ground truth labels or prior knowledge for anomalies is hard in real-world systems, which often lead to limited or no anomaly labels in the dataset. Second, anomalies in large systems usually occur in a collective manner due to the underlying dependency structure among devices or sensors. Lastly, real-time anomaly detection for high-dimensional data requires efficient algorithms that are capable of handling different types of data (i.e. continuous and discrete). We propose a correlation structure-based collective anomaly detection (CSCAD) model for high-dimensional anomaly detection problem in large systems, which is also generalizable to semi-supervised or supervised settings. Our framework utilize graph convolutional network combining a variational autoencoder to jointly exploit the feature space correlation and reconstruction deficiency of samples to perform anomaly detection. We propose an extended mutual information (EMI) metric to mine the internal correlation structure among different data features, which enhances the data reconstruction capability of CSCAD. The reconstruction loss and latent standard deviation vector of a sample obtained from reconstruction network can be perceived as two natural anomalous degree measures. An anomaly discriminating network can then be trained using low anomalous degree samples as positive samples, and high anomalous degree samples as negative samples. Experimental results on five public datasets demonstrate that our approach consistently outperforms all the competing baselines.
Author	Zhan, Xianyuan Qin, Huiling Zheng, Yu
Author_xml	– sequence: 1 givenname: Huiling orcidid: 0000-0002-4045-6091 surname: Qin fullname: Qin, Huiling email: orekinana@gmail.com organization: Xidian University and JD iCity, JD Technology, Beijing, China – sequence: 2 givenname: Xianyuan orcidid: 0000-0002-3683-0554 surname: Zhan fullname: Zhan, Xianyuan email: zhanxianyuan@gmail.com organization: Institute for AI Industry Research (AIR), Tsinghua University, Beijing, China – sequence: 3 givenname: Yu orcidid: 0000-0002-5224-4344 surname: Zheng fullname: Zheng, Yu email: msyuzheng@outlook.com organization: Xidian University and JD iCity, JD Technology, Beijing, China
BookMark	eNp9kMtOwzAQRS1UJErhAxCbSKxT7LEdx-xKWh6iEkIt68hJJlKqPIrtIvr3JLRiwYLVjObeM6O552TUdi0ScsXolDGqb9cv88UUKMCUMylYFJ2QMZMyDoFpNup7KlgouFBn5Ny5DaU0VjEbk7dklczmd0HSWYu18VXXBitvd7nfWQzvjcOi1-oac199YjBru8bU-2COfpj05qrt9WZb41ew2juPzQU5LU3t8PJYJ-T9YbFOnsLl6-NzMluGOWjuQ1PIiHFRoIYyYyozUMhcYKR1ITMFqDMeZ5JqCahEZAA5CDBU5Qp0qRjwCbk57N3a7mOHzqebbmfb_mQKSsuYQqR571IHV2475yyWaV75nze9NVWdMpoO-aVDfumQX3rMryfZH3Jrq8bY_b_M9YGpEPHXr5WglMf8G8aGfCk
CODEN	ITKEEH
CitedBy_id	crossref_primary_10_4018_IJDWM_327363 crossref_primary_10_1109_TKDE_2024_3488375 crossref_primary_10_1016_j_resconrec_2023_107239 crossref_primary_10_1109_TKDE_2023_3250523 crossref_primary_10_3389_frai_2024_1508821 crossref_primary_10_1155_2023_2332669 crossref_primary_10_1016_j_mlwa_2025_100728
Cites_doi	10.1007/978-3-319-48057-2_9 10.1007/978-3-030-10925-7_1 10.1109/ICDM.2008.17 10.1145/3191786 10.1145/3178876.3185996 10.1109/65.283931 10.1109/IJCNN.2016.7727242 10.1007/s10955-006-9131-x 10.1016/j.jnca.2015.11.016 10.1007/978-3-030-10925-7_11 10.1109/TBDATA.2020.2991008 10.1609/aaai.v33i01.33015167 10.1145/3292500.3330672 10.1145/3292500.3330871 10.1214/aoms/1177704472 10.1109/ICASSP.2008.4518376 10.1145/335191.335388 10.1145/1970392.1970395 10.1145/3292500.3330932 10.1007/s10994-020-05877-5 10.1145/3097983.3098052 10.1016/j.sigpro.2013.12.026 10.1007/978-3-319-59050-9_12 10.1145/1150402.1150459 10.1145/3097983.3098144 10.1145/3292500.3330680 10.1145/3292500.3330748 10.3156/jsoft.29.5_177_2 10.1145/2820783.2820813 10.1007/springerreference_205676 10.1109/ICIP.2001.958946 10.1145/1541880.1541882
ContentType	Journal Article
Copyright	Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2023
Copyright_xml	– notice: Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2023
DBID	97E RIA RIE AAYXX CITATION 7SC 7SP 8FD JQ2 L7M L~C L~D
DOI	10.1109/TKDE.2022.3154166
DatabaseName	IEEE Xplore (IEEE) IEEE All-Society Periodicals Package (ASPP) 1998–Present IEEE Xplore CrossRef Computer and Information Systems Abstracts Electronics & Communications Abstracts Technology Research Database ProQuest Computer Science Collection Advanced Technologies Database with Aerospace Computer and Information Systems Abstracts Academic Computer and Information Systems Abstracts Professional
DatabaseTitle	CrossRef Technology Research Database Computer and Information Systems Abstracts – Academic Electronics & Communications Abstracts ProQuest Computer Science Collection Computer and Information Systems Abstracts Advanced Technologies Database with Aerospace Computer and Information Systems Abstracts Professional
DatabaseTitleList	Technology Research Database
Database_xml	– sequence: 1 dbid: RIE name: IEEE Electronic Library Online url: https://ieeexplore.ieee.org/ sourceTypes: Publisher
DeliveryMethod	fulltext_linktorsrc
Discipline	Engineering Computer Science
EISSN	1558-2191
EndPage	4645
ExternalDocumentID	10_1109_TKDE_2022_3154166 9740038
Genre	orig-research
GrantInformation_xml	– fundername: National Key R&D Program of China grantid: 2019YFB2101801 – fundername: Public Service Platform for Industrial Technology grantid: 2021-0166-1-1 – fundername: National Natural Science Foundation of China grantid: 62076191 funderid: 10.13039/501100001809
GroupedDBID	-~X .DC 0R~ 29I 4.4 5GY 6IK 97E AAJGR AARMG AASAJ AAWTH ABAZT ABQJQ ABVLG ACGFO ACIWK AENEX AGQYO AHBIQ AKJIK AKQYR ALMA_UNASSIGNED_HOLDINGS ASUFR ATWAV BEFXN BFFAM BGNUA BKEBE BPEOZ CS3 DU5 EBS EJD F5P HZ~ IEDLZ IFIPE IPLJI JAVBF LAI M43 MS~ O9- OCL P2P PQQKQ RIA RIE RNS RXW TAE TN5 UHB 1OL 5VS 9M8 AAYXX ABFSI AETIX AGSQL AI. AIBXA ALLEH CITATION E.L H~9 ICLAB IFJZH RNI RZB TAF VH1 7SC 7SP 8FD JQ2 L7M L~C L~D
ID	FETCH-LOGICAL-c293t-ad56134de92fb17ba2d5c4e699d5b72e9b38b50952e746a2e3242a07c729f7123
IEDL.DBID	RIE
ISICitedReferencesCount	9
ISICitedReferencesURI	http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000964880800019&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
ISSN	1041-4347
IngestDate	Mon Jun 30 04:14:17 EDT 2025 Sat Nov 29 02:36:05 EST 2025 Tue Nov 18 21:01:00 EST 2025 Wed Aug 27 02:14:18 EDT 2025
IsPeerReviewed	true
IsScholarly	true
Issue	5
Language	English
License	https://ieeexplore.ieee.org/Xplorehelp/downloads/license-information/IEEE.html https://doi.org/10.15223/policy-029 https://doi.org/10.15223/policy-037
LinkModel	DirectLink
MergedId	FETCHMERGED-LOGICAL-c293t-ad56134de92fb17ba2d5c4e699d5b72e9b38b50952e746a2e3242a07c729f7123
Notes	ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 14
ORCID	0000-0002-5224-4344 0000-0002-4045-6091 0000-0002-3683-0554
PQID	2795802693
PQPubID	85438
PageCount	12
ParticipantIDs	crossref_citationtrail_10_1109_TKDE_2022_3154166 crossref_primary_10_1109_TKDE_2022_3154166 ieee_primary_9740038 proquest_journals_2795802693
PublicationCentury	2000
PublicationDate	2023-05-01
PublicationDateYYYYMMDD	2023-05-01
PublicationDate_xml	– month: 05 year: 2023 text: 2023-05-01 day: 01
PublicationDecade	2020
PublicationPlace	New York
PublicationPlace_xml	– name: New York
PublicationTitle	IEEE transactions on knowledge and data engineering
PublicationTitleAbbrev	TKDE
PublicationYear	2023
Publisher	IEEE The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher_xml	– name: IEEE – name: The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
References	ref13 ref12 ref34 ref15 ref37 ref14 ref31 Ikeda (ref19) 2018 ref30 ref33 ref10 Kingma (ref35) 2014 ref2 ref1 ref17 ref39 ref16 ref38 Dua (ref11) 2017 Zong (ref5) 2018 Defferrard (ref36) 2016 Zhai (ref4) 2016 Li (ref32) 2018 ref24 ref23 ref26 ref25 ref20 ref22 ref21 An (ref18) 2015; 2 ref28 ref27 ref29 ref8 ref7 ref9 ref3 ref6 ref40
References_xml	– ident: ref7 doi: 10.1007/978-3-319-48057-2_9 – ident: ref31 doi: 10.1007/978-3-030-10925-7_1 – ident: ref39 doi: 10.1109/ICDM.2008.17 – ident: ref10 doi: 10.1145/3191786 – ident: ref20 doi: 10.1145/3178876.3185996 – ident: ref1 doi: 10.1109/65.283931 – year: 2018 ident: ref32 article-title: Anomaly detection with generative adversarial networks for multivariate time series – ident: ref8 doi: 10.1109/IJCNN.2016.7727242 – ident: ref34 doi: 10.1007/s10955-006-9131-x – ident: ref12 doi: 10.1016/j.jnca.2015.11.016 – ident: ref29 doi: 10.1007/978-3-030-10925-7_11 – ident: ref15 doi: 10.1109/TBDATA.2020.2991008 – ident: ref6 doi: 10.1609/aaai.v33i01.33015167 – ident: ref23 doi: 10.1145/3292500.3330672 – ident: ref25 doi: 10.1145/3292500.3330871 – ident: ref16 doi: 10.1214/aoms/1177704472 – ident: ref17 doi: 10.1109/ICASSP.2008.4518376 – ident: ref38 doi: 10.1145/335191.335388 – ident: ref3 doi: 10.1145/1970392.1970395 – year: 2018 ident: ref19 article-title: Estimation of dimensions contributing to detected anomalies with variational autoencoders – ident: ref21 doi: 10.1145/3292500.3330932 – ident: ref37 doi: 10.1007/s10994-020-05877-5 – ident: ref28 doi: 10.1145/3097983.3098052 – ident: ref14 doi: 10.1016/j.sigpro.2013.12.026 – volume-title: Proc. Int. Conf. Learn. Representations year: 2018 ident: ref5 article-title: Deep autoencoding gaussian mixture model for unsupervised anomaly detection – ident: ref33 doi: 10.1007/978-3-319-59050-9_12 – ident: ref2 doi: 10.1145/1150402.1150459 – year: 2017 ident: ref11 article-title: UCI machine learning repository – ident: ref24 doi: 10.1145/3097983.3098144 – ident: ref26 doi: 10.1145/3292500.3330680 – start-page: 3844 volume-title: Proc. Adv. Neural Inf. Process. Syst. year: 2016 ident: ref36 article-title: Convolutional neural networks on graphs with fast localized spectral filtering – volume: 2 start-page: 1 year: 2015 ident: ref18 article-title: Variational autoencoder based anomaly detection using reconstruction probability publication-title: Special Lecture IE – ident: ref22 doi: 10.1145/3292500.3330748 – year: 2014 ident: ref35 article-title: Auto-encoding variational bayes – ident: ref30 doi: 10.3156/jsoft.29.5_177_2 – ident: ref9 doi: 10.1145/2820783.2820813 – ident: ref27 doi: 10.1007/springerreference_205676 – ident: ref40 doi: 10.1109/ICIP.2001.958946 – ident: ref13 doi: 10.1145/1541880.1541882 – start-page: 1100 volume-title: Proc. Int. Conf. Mach. Learn. year: 2016 ident: ref4 article-title: Deep structured energy based models for anomaly detection
SSID	ssj0008781
Score	2.451345
Snippet	Detecting anomalies in large complex systems is a critical and challenging task. The difficulties arise from several aspects. First, collecting ground truth...
SourceID	proquest crossref ieee
SourceType	Aggregation Database Enrichment Source Index Database Publisher
StartPage	4634
SubjectTerms	Algorithms Anomalies Anomaly detection complex system Complex systems Correlation correlation mining Data models Datasets Feature extraction Labels Loss measurement Reconstruction Sensors unsupervised learning urban computing variational autoencoder
Title	CSCAD: Correlation Structure-Based Collective Anomaly Detection in Complex System
URI	https://ieeexplore.ieee.org/document/9740038 https://www.proquest.com/docview/2795802693
Volume	35
WOSCitedRecordID	wos000964880800019&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
hasFullText	1
inHoldings	1
isFullTextHit
isPrint
journalDatabaseRights	– providerCode: PRVIEE databaseName: IEEE Electronic Library Online customDbUrl: eissn: 1558-2191 dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0008781 issn: 1041-4347 databaseCode: RIE dateStart: 19890101 isFulltext: true titleUrlDefault: https://ieeexplore.ieee.org/ providerName: IEEE
link	http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV3dS8MwED_m8EEfnG6K0yl58EmM69KPNL7NfSAoQ9mEvZU2yWAwO9mH6H_vJe2Kogi-FZrQcr_m7n7N5X4AF7LFJa4_ST3fiak3CQVNlB9S11NMyVj7IrAHhR_4YBCOx-KxBFfFWRittS0-09fm0u7lq7lcm19lTcx9zU7WFmxxzrOzWoXXDbkVJEV2gZzI9Xi-g9lyRHN03-0hE2QMCaqPCUjwLQZZUZUfntiGl37lfy-2D3t5GknaGe4HUNJpFSobiQaSr9gq7H7pN1iDp84QLX9DOkaSIyuCI0PbQHa90PQWA5oi9k-CdYKknc5f4tkH6eqVLdhKyTQl5hkz_U6yVueH8NzvjTp3NNdUoBID-4rGyjAGT2nBJkmLJzFTvvR0IITyE860SNwwwSTCZ5p7Qcy0Sbhih0tMwiccw9wRlNN5qo-B4I0wQYeBeEqPTUTMAyfB7NHFiRINXQdnY-VI5g3Hje7FLLLEwxGRASYywEQ5MHW4LKa8Zt02_hpcM0gUA3MQ6tDYQBnl63EZMS78EOmmcE9-n3UKO0ZIPitlbEAZba_PYFu-rabLxbn91D4B5gLOkg
linkProvider	IEEE
linkToHtml	http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1bS8MwFD7MC6gPXjbFec2DT2JdlyZN45tuE8U5FCfsrbRJBsLsxG2i_96TtBuKIvhWaEJLvuSc70tOzgE4UnWhcP0pj3E_8Vg_kl6qeeQFTFOtEsNl6C4Kt0WnE_V68q4EJ7O7MMYYF3xmTu2jO8vXQzWxW2U15L72JGsOFjhjtJ7f1prZ3Ui4kqSoL1AVBUwUZ5h1X9a6N80WakFKUaJypCDhNy_kyqr8sMXOwVyu_e_X1mG1IJLkPEd-A0omK8PatEgDKdZsGVa-ZByswH3jAcf-jDRsUY48DI48uBSyk1fjXaBL08TtJTgzSM6z4XMy-CBNM3YhWxl5yoj9xsC8kzzZ-SY8Xra6jSuvqKrgKXTtYy_RVjMwbSTtp3WRJlRzxUwopeapoEamQZQijeDUCBYm1FjKlfhCIQ3vC3R0WzCfDTOzDQRfRCmaDERUMdqXiQj9FPljgB0VDnQV_Okox6pIOW4rXwxiJz18GVtgYgtMXABTheNZl5c838ZfjSsWiVnDAoQq7E2hjIsVOYqpkDxCwSmDnd97HcLSVfe2HbevOze7sGzLyueBjXswjziYfVhUb-On0euBm3af0RTR2Q
openUrl	ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=CSCAD%3A+Correlation+Structure-Based+Collective+Anomaly+Detection+in+Complex+System&rft.jtitle=IEEE+transactions+on+knowledge+and+data+engineering&rft.au=Qin%2C+Huiling&rft.au=Zhan%2C+Xianyuan&rft.au=Zheng%2C+Yu&rft.date=2023-05-01&rft.pub=The+Institute+of+Electrical+and+Electronics+Engineers%2C+Inc.+%28IEEE%29&rft.issn=1041-4347&rft.eissn=1558-2191&rft.volume=35&rft.issue=5&rft.spage=4634&rft_id=info:doi/10.1109%2FTKDE.2022.3154166&rft.externalDBID=NO_FULL_TEXT
thumbnail_l	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=1041-4347&client=summon
thumbnail_m	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=1041-4347&client=summon
thumbnail_s	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=1041-4347&client=summon