Advanced memory forensics for malware classification with deep learning algorithms - Odeh, Ammar; Taleb, Anas Abu; Alhajahjeh, Tareq; Navarro, Francisco (Cluster computing, 2025)
The growing complexity of malware, especially polymorphic and obfuscated variants, has exposed significant limitations in traditional detection methods. This study addresses these challenges using memory forensics to detect and classify malware through deep learning algorithms. Memory-based features, including memory pages, threads, open files, user sessions, system calls, and kernel modules, were extracted from memory dumps using the Volatility and Rekall frameworks. Three deep learning models—Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and Autoencoders—were applied to analyze the extracted features. The dataset was divided into ten subsets using 10-fold cross-validation to ensure robustness and prevent overfitting. The models’ performance was evaluated using accuracy, precision, recall, and F1-score metrics. The results show that CNNs and RNNs consistently outperformed Autoencoders, with CNNs achieving the highest accuracy of 97.8%. These findings demonstrate the superior effectiveness of CNNs and RNNs in detecting malware using memory-based data. This research establishes deep learning algorithms, particularly CNNs and RNNs, as powerful tools for malware detection in cybersecurity. In conclusion, this study contributes to ongoing efforts to enhance malware detection systems by leveraging memory forensics and deep learning. Future work will explore additional feature extraction techniques and hybrid model architectures to improve detection accuracy further and reduce false positives.