AcLog: An Approach to Detecting Anomalies from System Logs with Active Learning

Log-based anomaly detection is an essential aspect of maintaining software reliability, particularly in the context of microservice systems. However, existing log-based anomaly detection approaches rely on historical anomalous labeled data or require huge labeling efforts. This makes existing log-ba...

Full description

Saved in:
Bibliographic Details
Published in:Proceedings (IEEE International Conference on Web Services. Online) pp. 436 - 443
Main Authors: Duan, Chiming, Jia, Tong, Li, Ying, Huang, Gang
Format: Conference Proceeding
Language:English
Published: IEEE 01.07.2023
Subjects:
Active learning
Anomaly detection
Data models
Deep Learning
Human Feedback
Labeling
Log analysis
Microservice architectures
Software reliability
Training
Training data
Web services
ISSN:2836-3868
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Log-based anomaly detection is an essential aspect of maintaining software reliability, particularly in the context of microservice systems. However, existing log-based anomaly detection approaches rely on historical anomalous labeled data or require huge labeling efforts. This makes existing log-based anomaly detection approaches inefficient. In this paper, we propose AcLog, a novel anomaly detection approach that incorporates human knowledge to enhance model ability based on the framework of deep active learning. It incorporates an unsupervised model to learn from normal log data rather than historical anomalous labeled data and leverages active learning to incorporate human knowledge as a golden signal to augment the quality of training log data. Experiment results on three open log datasets and one log dataset collected from a real-world microservice system show that our approach improves over 7% F1-score with 5% labeled training data on average.
ISSN:2836-3868
DOI:10.1109/ICWS60048.2023.00062