An online log template extraction method based on hierarchical clustering

The raw log messages record extremely rich system, network, and application running dynamic information that is a good data source for abnormal detection. Log template extraction is an important prerequisite for log sequence anomaly detection. The problems of the existing log template extraction met...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:EURASIP journal on wireless communications and networking Jg. 2019; H. 1; S. 1 - 12
Hauptverfasser: Yang, Ruipeng, Qu, Dan, Qian, Yekui, Dai, Yusheng, Zhu, Shaowei
Format: Journal Article
Sprache:Englisch
Veröffentlicht: Cham Springer International Publishing 28.05.2019
Springer Nature B.V
SpringerOpen
Schlagworte:
Algorithms
Anomalies
Cluster analysis
Clustering
Communications Engineering
Distributed Big Data Processing in SCADA Communication Networks
Engineering
Information Systems Applications (incl.Internet)
Log anomaly detection
Log template extraction
Messages
Networks
Online hierarchical clustering
Online vectorization
Signal,Image and Speech Processing
Log template extraction
Online vectorization
Log anomaly detection
Online hierarchical clustering
ISSN:1687-1499, 1687-1472, 1687-1499
Online-Zugang:Volltext
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Beschreibung
Zusammenfassung:The raw log messages record extremely rich system, network, and application running dynamic information that is a good data source for abnormal detection. Log template extraction is an important prerequisite for log sequence anomaly detection. The problems of the existing log template extraction methods are mostly offline, and the few online methods have insufficient F1-score in multi-source log data. In view of the shortcomings of the existing methods, an online log template extraction method called LogOHC is proposed. Firstly, the raw log messages are preprocessed, and the word distributed representation (word2vec) is used to vectorize the log messages online. Then, the online hierarchical clustering algorithm is applied, and finally, log templates are generated. The experimental analysis shows that LogOHC has a higher F1-score than the existing log template extraction methods, is suitable for multi-source log data sets, and has a shorter single-step execution time, which can meet the requirements of online real-time processing.
Bibliographie:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:1687-1499
1687-1472
1687-1499
DOI:10.1186/s13638-019-1430-4