AutoLog: Anomaly detection by deep autoencoding of system logs

The use of system logs for detecting and troubleshooting anomalies of production systems has been known since the early days of computers. In spite of the advances in the area, the analysis of log files emitted by real-life systems poses many peculiar challenges. Up-to-date tools, such as log manage...

Full description

Saved in:
Bibliographic Details
Published in:Expert systems with applications Vol. 191; p. 116263
Main Authors: Catillo, Marta, Pecchia, Antonio, Villano, Umberto
Format: Journal Article
Language:English
Published: New York Elsevier Ltd 01.04.2022
Elsevier BV
Subjects:
Anomalies
Anomaly detection
Autoencoder
Comparative studies
Cybersecurity
Data logging
Decision trees
Deep learning
Information management
Standard data
System logs
Troubleshooting
Deep learning
Cybersecurity
Autoencoder
System logs
Anomaly detection
ISSN:0957-4174, 1873-6793
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:The use of system logs for detecting and troubleshooting anomalies of production systems has been known since the early days of computers. In spite of the advances in the area, the analysis of log files emitted by real-life systems poses many peculiar challenges. Up-to-date tools, such as log management and Security Information and Event Management (SIEM) products, capitalize on standard data formats, logging protocols and dictionaries of threat signatures, which hardly fit to logs of industrial and proprietary systems. This paper addresses the analysis of logs emitted by computer systems with a focus on anomaly detection. The proposed approach, named AutoLog, consists in sampling the logs at regular intervals and to compute numeric scores. Scores collected under normative operations are used to train a semi-supervised deep autoencoder, which serves as a baseline to classify future scores. The approach is not constrained by the structure of underlying logs and does not need for anomalies at training time. The results obtained in detecting anomalies of two industrial systems and the public BG/L and Hadoop datasets widely used as benchmarks, indicate that the recall of AutoLog ranges between 0.96 and 0.99, while the precision is within 0.93 and 0.98. A comparative study with isolation forest, one-class SVM, decision tree, vanilla autoencoder and variational autoencoder is conducted to demonstrate the validity of the proposal. •A semi-supervised learning technique for anomaly detection.•Automatic knowledge extraction from system logs.•Deep autoencoding and non-linear data transformations.•Measurements with real-life computer systems and logs.
Bibliography:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:0957-4174
1873-6793
DOI:10.1016/j.eswa.2021.116263