A Hybrid Deep Learning Approach for Detecting Anomalies in Real-Time Data Streams

A crucial problem in many fields, such as cyber-security, financial fraud detection, industrial monitoring, and healthcare, is anomaly identification in real-time data streams. Conventional anomaly detection approaches, such machine learning models and statistical techniques, have trouble processing...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:International Conference for Emerging Technology (Online) S. 1 - 6
Hauptverfasser: Mishra, Aditya Mohan, Yadav, Daksh, Shakya, Ananmaya, Jayesh, Vagh, Bala, Er. Neetu
Format: Tagungsbericht
Sprache:Englisch
Veröffentlicht: IEEE 23.05.2025
Schlagworte:
Adaptive Thresholding
Anomaly detection
Autoen-coders
Concept drift
Cybersecurity
Deep learning
Dimensionality reduction
Feature extraction
Fraud
Fraud Detection
Generative adversarial networks
Generative Adversarial Networks (GANs)
Low latency communication
Low-Latency Processing
Real-time systems
Scalable Anomaly Detection
Self-Supervised Learning
Training
ISBN:9798331518738
ISSN:2996-4490
Online-Zugang:Volltext
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Beschreibung
Zusammenfassung:A crucial problem in many fields, such as cyber-security, financial fraud detection, industrial monitoring, and healthcare, is anomaly identification in real-time data streams. Conventional anomaly detection approaches, such machine learning models and statistical techniques, have trouble processing high-dimensional data, adjusting to concept drift, and sustaining low-latency performance in real-time settings. In order to overcome these obstacles, this study suggests a scalable and reliable anomaly detection framework that makes use of deep learning methods, specifically Generative Adversarial Networks (GANs) for anomaly generation and detection and autoencoders (AEs) for dimensionality reduction. There are two main parts to the suggested framework. In order to reduce computational complexity while maintaining important data features, an autoencoder-based feature extraction module first learns compact representations of normal data. High reconstruction error data points are marked as possible abnormalities. Second, by training a discriminator to discern between generated and genuine normal samples, and training a generator to replicate normal data distributions, a GAN-based anomaly detection model improves the detection process even more. An anomaly is defined as any real-time data point that substantially differs from the learnt distribution. The system uses model compression methods like quantisation and pruning, distributed processing using Apache Spark and Kafka, and an adaptive learning mechanism to dynamically tackle concept drift in order to maximise scalability and guarantee real-time performance. Experiments on benchmark datasets such as KDD99, UNSW-NB15, and SWaT show that our method achieves low false positive rates, high detection accuracy (AUC-ROC 0.95), and real-time inference speeds (50ms per data point).For real-world applications where prompt anomaly identification is essential, this research offers a scalable, high-performance anomaly detection system. To further improve the detection system's resilience, future research will investigate hybrid deep learning models that combine reinforcement learning and transformers.
ISBN:9798331518738
ISSN:2996-4490
DOI:10.1109/INCET64471.2025.11140026