A Hybrid Deep Learning Approach for Detecting Anomalies in Real-Time Data Streams
A crucial problem in many fields, such as cyber-security, financial fraud detection, industrial monitoring, and healthcare, is anomaly identification in real-time data streams. Conventional anomaly detection approaches, such machine learning models and statistical techniques, have trouble processing...
Gespeichert in:
| Veröffentlicht in: | International Conference for Emerging Technology (Online) S. 1 - 6 |
|---|---|
| Hauptverfasser: | , , , , |
| Format: | Tagungsbericht |
| Sprache: | Englisch |
| Veröffentlicht: |
IEEE
23.05.2025
|
| Schlagworte: | |
| ISBN: | 9798331518738 |
| ISSN: | 2996-4490 |
| Online-Zugang: | Volltext |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| Abstract | A crucial problem in many fields, such as cyber-security, financial fraud detection, industrial monitoring, and healthcare, is anomaly identification in real-time data streams. Conventional anomaly detection approaches, such machine learning models and statistical techniques, have trouble processing high-dimensional data, adjusting to concept drift, and sustaining low-latency performance in real-time settings. In order to overcome these obstacles, this study suggests a scalable and reliable anomaly detection framework that makes use of deep learning methods, specifically Generative Adversarial Networks (GANs) for anomaly generation and detection and autoencoders (AEs) for dimensionality reduction. There are two main parts to the suggested framework. In order to reduce computational complexity while maintaining important data features, an autoencoder-based feature extraction module first learns compact representations of normal data. High reconstruction error data points are marked as possible abnormalities. Second, by training a discriminator to discern between generated and genuine normal samples, and training a generator to replicate normal data distributions, a GAN-based anomaly detection model improves the detection process even more. An anomaly is defined as any real-time data point that substantially differs from the learnt distribution. The system uses model compression methods like quantisation and pruning, distributed processing using Apache Spark and Kafka, and an adaptive learning mechanism to dynamically tackle concept drift in order to maximise scalability and guarantee real-time performance. Experiments on benchmark datasets such as KDD99, UNSW-NB15, and SWaT show that our method achieves low false positive rates, high detection accuracy (AUC-ROC 0.95), and real-time inference speeds (50ms per data point).For real-world applications where prompt anomaly identification is essential, this research offers a scalable, high-performance anomaly detection system. To further improve the detection system's resilience, future research will investigate hybrid deep learning models that combine reinforcement learning and transformers. |
|---|---|
| AbstractList | A crucial problem in many fields, such as cyber-security, financial fraud detection, industrial monitoring, and healthcare, is anomaly identification in real-time data streams. Conventional anomaly detection approaches, such machine learning models and statistical techniques, have trouble processing high-dimensional data, adjusting to concept drift, and sustaining low-latency performance in real-time settings. In order to overcome these obstacles, this study suggests a scalable and reliable anomaly detection framework that makes use of deep learning methods, specifically Generative Adversarial Networks (GANs) for anomaly generation and detection and autoencoders (AEs) for dimensionality reduction. There are two main parts to the suggested framework. In order to reduce computational complexity while maintaining important data features, an autoencoder-based feature extraction module first learns compact representations of normal data. High reconstruction error data points are marked as possible abnormalities. Second, by training a discriminator to discern between generated and genuine normal samples, and training a generator to replicate normal data distributions, a GAN-based anomaly detection model improves the detection process even more. An anomaly is defined as any real-time data point that substantially differs from the learnt distribution. The system uses model compression methods like quantisation and pruning, distributed processing using Apache Spark and Kafka, and an adaptive learning mechanism to dynamically tackle concept drift in order to maximise scalability and guarantee real-time performance. Experiments on benchmark datasets such as KDD99, UNSW-NB15, and SWaT show that our method achieves low false positive rates, high detection accuracy (AUC-ROC 0.95), and real-time inference speeds (50ms per data point).For real-world applications where prompt anomaly identification is essential, this research offers a scalable, high-performance anomaly detection system. To further improve the detection system's resilience, future research will investigate hybrid deep learning models that combine reinforcement learning and transformers. |
| Author | Shakya, Ananmaya Yadav, Daksh Bala, Er. Neetu Mishra, Aditya Mohan Jayesh, Vagh |
| Author_xml | – sequence: 1 givenname: Aditya Mohan surname: Mishra fullname: Mishra, Aditya Mohan email: adityammishra1110@gmail.com organization: Chandigarh University,Dept. of Computer Science – sequence: 2 givenname: Daksh surname: Yadav fullname: Yadav, Daksh email: dakshyadav0987@gmail.com organization: Chandigarh University,Dept. of Computer Science – sequence: 3 givenname: Ananmaya surname: Shakya fullname: Shakya, Ananmaya email: ananmayashakya@gmail.com organization: Chandigarh University,Dept. of Computer Science – sequence: 4 givenname: Vagh surname: Jayesh fullname: Jayesh, Vagh email: jayeshvagh955@gmail.com organization: Chandigarh University,Dept. of Computer Science – sequence: 5 givenname: Er. Neetu surname: Bala fullname: Bala, Er. Neetu email: neetubala211@gmail.in organization: Chandigarh University,Dept. of Computer Science |
| BookMark | eNpVkMlOwzAURc0kUUr-gIV_IMVT7LxllBZaKQIB2Vd28gJGmeRk078nYliwuVc6R7qLe0Mu-6FHQihnG84Z3B-e8l2plTJ8I5hIFsgVY0KfkQgMpFLyhKdG8HOyEgA6VgrYxT8n02sSTdMnY0wKJg0XK_KS0f3JBV_TLeJIC7Sh9_07zcYxDLb6oM0QFjVjNX_jfuhs63GivqevaNu49B3SrZ0tfZsD2m66JVeNbSeMfntNyoddme_j4vnxkGdF7EHOcVULk6BUkGqnFTqmEZrK1YkDV-klwCijQKAygNKCQ9CCi7SWae2cauSa3P3MekQ8jsF3NpyOf6fIL85sVV4 |
| ContentType | Conference Proceeding |
| DBID | 6IE 6IL CBEJK RIE RIL |
| DOI | 10.1109/INCET64471.2025.11140026 |
| DatabaseName | IEEE Electronic Library (IEL) Conference Proceedings IEEE Xplore POP ALL IEEE Xplore All Conference Proceedings IEEE/IET Electronic Library IEEE Proceedings Order Plans (POP All) 1998-Present |
| DatabaseTitleList | |
| Database_xml | – sequence: 1 dbid: RIE name: IEEE/IET Electronic Library url: https://ieeexplore.ieee.org/ sourceTypes: Publisher |
| DeliveryMethod | fulltext_linktorsrc |
| EISBN | 9798331518721 9798331531034 |
| EISSN | 2996-4490 |
| EndPage | 6 |
| ExternalDocumentID | 11140026 |
| Genre | orig-research |
| GroupedDBID | 6IE 6IF 6IL 6IN ABLEC ADZIZ ALMA_UNASSIGNED_HOLDINGS BEFXN BFFAM BGNUA BKEBE BPEOZ CBEJK CHZPO IEGSK OCL RIE RIL |
| ID | FETCH-LOGICAL-i93t-cd275e34986b64eb06e9fcbd5b9bc6b9b9747492e479e3a9be962128d38dbb4f3 |
| IEDL.DBID | RIE |
| ISBN | 9798331518738 |
| IngestDate | Wed Sep 10 07:40:44 EDT 2025 |
| IsPeerReviewed | false |
| IsScholarly | false |
| Language | English |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-i93t-cd275e34986b64eb06e9fcbd5b9bc6b9b9747492e479e3a9be962128d38dbb4f3 |
| PageCount | 6 |
| ParticipantIDs | ieee_primary_11140026 |
| PublicationCentury | 2000 |
| PublicationDate | 2025-May-23 |
| PublicationDateYYYYMMDD | 2025-05-23 |
| PublicationDate_xml | – month: 05 year: 2025 text: 2025-May-23 day: 23 |
| PublicationDecade | 2020 |
| PublicationTitle | International Conference for Emerging Technology (Online) |
| PublicationTitleAbbrev | INCET |
| PublicationYear | 2025 |
| Publisher | IEEE |
| Publisher_xml | – name: IEEE |
| SSID | ssj0003203712 |
| Score | 1.9094332 |
| Snippet | A crucial problem in many fields, such as cyber-security, financial fraud detection, industrial monitoring, and healthcare, is anomaly identification in... |
| SourceID | ieee |
| SourceType | Publisher |
| StartPage | 1 |
| SubjectTerms | Adaptive Thresholding Anomaly detection Autoen-coders Concept drift Cybersecurity Deep learning Dimensionality reduction Feature extraction Fraud Fraud Detection Generative adversarial networks Generative Adversarial Networks (GANs) Low latency communication Low-Latency Processing Real-time systems Scalable Anomaly Detection Self-Supervised Learning Training |
| Title | A Hybrid Deep Learning Approach for Detecting Anomalies in Real-Time Data Streams |
| URI | https://ieeexplore.ieee.org/document/11140026 |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV09b8IwELUK6tCprUrVb3noaghx4o8RlSK6IFoxsCE7PlcMBAQBqf--PhOoOnToElmOYkVn2e_O5_eOkOfEgxQ2LKTcgGJhwyuYNVoxL4A7L4w21sViE3I0UtOpHtdk9ciFAYB4-Qza2Iy5fLcstnhU1gnrMsOgoUEaUoo9Wet4oMJTVJ-Len1SK84DlCnJ1eH2TqI7b1jtJeC_xLgwzduH4X4VVom4Mjj_5x9dkNYPQ4-Oj9hzSU6gvCLvPTr8QgYW7QOsaC2d-kl7tW44DQ5qeIVpg9hdLhfBC4cNnZf0IziMDPkgtG8qQzFZbRabFpkMXicvQ1aXTGBzzStWuFTmwDOthBUZ2ESA9oV1udW2EOGB0UOmU8ikBm60BS0CdinHlbM28_yaNMtlCTeE5rLouvCVD3tC1jXOeI3SbakyKrHC-lvSQnPMVntRjNnBEnd_9N-TMzQ6Jt5T_kCa1XoLj-S02FXzzfopTuU3ZaKc2w |
| linkProvider | IEEE |
| linkToHtml | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1NTwIxEG0UTfSkRozf9uB1Ydnu9uNIRAMRCRoO3Ei7nRoOLAQWE_-9nbpgPHjw0jTdbNO0aWem0_ceIfexA8GN30iZBhn5Ay-PjFYychyYdVwrbWwQmxCDgRyP1bACqwcsDACEx2fQwGrI5dt5vsarsqbflykGDbtkD6WzKrjW9kqFJcg_Fxj7hJKMeWMmBZOb9zuxavZQ78V7AAIjwyRrbDr8Ja0SLMvT0T_HdEzqPxg9OtxanxOyA8UpeW3T7idisGgHYEEr8tR32q6Yw6l3Uf0nTByE5mI-8344rOi0oG_eZYwQEUI7utQU09V6tqqT0dPj6KEbVaIJ0VSxMsptIjJgqZLc8BRMzEG53NjMKJNzX2D8kKoEUqGAaWVAcW-9pGXSGpM6dkZqxbyAc0Izkbes_8v5UyFtaaudQvK2RGoZG27cBanjdEwW37QYk81MXP7RfkcOuqOX_qTfGzxfkUNcAEzDJ-ya1MrlGm7Ifv5RTlfL27CsX8a0oCQ |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=proceeding&rft.title=International+Conference+for+Emerging+Technology+%28Online%29&rft.atitle=A+Hybrid+Deep+Learning+Approach+for+Detecting+Anomalies+in+Real-Time+Data+Streams&rft.au=Mishra%2C+Aditya+Mohan&rft.au=Yadav%2C+Daksh&rft.au=Shakya%2C+Ananmaya&rft.au=Jayesh%2C+Vagh&rft.date=2025-05-23&rft.pub=IEEE&rft.isbn=9798331518738&rft.eissn=2996-4490&rft.spage=1&rft.epage=6&rft_id=info:doi/10.1109%2FINCET64471.2025.11140026&rft.externalDocID=11140026 |
| thumbnail_l | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=9798331518738/lc.gif&client=summon&freeimage=true |
| thumbnail_m | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=9798331518738/mc.gif&client=summon&freeimage=true |
| thumbnail_s | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=9798331518738/sc.gif&client=summon&freeimage=true |