An online log template extraction method based on hierarchical clustering

The raw log messages record extremely rich system, network, and application running dynamic information that is a good data source for abnormal detection. Log template extraction is an important prerequisite for log sequence anomaly detection. The problems of the existing log template extraction met...

Celý popis

Uložené v:
Podrobná bibliografia
Vydané v:EURASIP journal on wireless communications and networking Ročník 2019; číslo 1; s. 1 - 12
Hlavní autori: Yang, Ruipeng, Qu, Dan, Qian, Yekui, Dai, Yusheng, Zhu, Shaowei
Médium: Journal Article
Jazyk:English
Vydavateľské údaje: Cham Springer International Publishing 28.05.2019
Springer Nature B.V
SpringerOpen
Predmet:
Algorithms
Anomalies
Cluster analysis
Clustering
Communications Engineering
Distributed Big Data Processing in SCADA Communication Networks
Engineering
Information Systems Applications (incl.Internet)
Log anomaly detection
Log template extraction
Messages
Networks
Online hierarchical clustering
Online vectorization
Signal,Image and Speech Processing
Log template extraction
Online vectorization
Log anomaly detection
Online hierarchical clustering
ISSN:1687-1499, 1687-1472, 1687-1499
On-line prístup:Získať plný text
Tagy: Pridať tag
Žiadne tagy, Buďte prvý, kto otaguje tento záznam!
Popis
Shrnutí:The raw log messages record extremely rich system, network, and application running dynamic information that is a good data source for abnormal detection. Log template extraction is an important prerequisite for log sequence anomaly detection. The problems of the existing log template extraction methods are mostly offline, and the few online methods have insufficient F1-score in multi-source log data. In view of the shortcomings of the existing methods, an online log template extraction method called LogOHC is proposed. Firstly, the raw log messages are preprocessed, and the word distributed representation (word2vec) is used to vectorize the log messages online. Then, the online hierarchical clustering algorithm is applied, and finally, log templates are generated. The experimental analysis shows that LogOHC has a higher F1-score than the existing log template extraction methods, is suitable for multi-source log data sets, and has a shorter single-step execution time, which can meet the requirements of online real-time processing.
Bibliografia:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:1687-1499
1687-1472
1687-1499
DOI:10.1186/s13638-019-1430-4