Anomaly Detection Algorithms on IBM InfoSphere Streams: Anomaly Detection for Data in Motion

This paper presents and shares excerpts from our implementation of near real-time anomaly detection algorithms on the IBM InfoSphere Streams platform. The purpose of this article is to: 1) Describe how to design and implement known anomaly detection algorithms on IBM InfoSphere Streams. 2) Present s...

Full description

Saved in:
Bibliographic Details
Published in:2012 IEEE 10th International Symposium on Parallel and Distributed Processing with Applications pp. 301 - 308
Main Authors: Yulevich, Y., Pyasik, A., Gorelik, L.
Format: Conference Proceeding
Language:English
Published: IEEE 01.07.2012
Subjects:
Algorithm design and analysis
Detection algorithms
Dictionaries
Kernel
Measurement
Principal component analysis
Vectors
ISBN:1467316318, 9781467316316
ISSN:2158-9178
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:This paper presents and shares excerpts from our implementation of near real-time anomaly detection algorithms on the IBM InfoSphere Streams platform. The purpose of this article is to: 1) Describe how to design and implement known anomaly detection algorithms on IBM InfoSphere Streams. 2) Present some performance optimization capabilities of IBM InfoSphere Streams platform and propose a method to use them in anomaly detection applications. 3) Present some IBM InfoSphere Streams best practices and describe how their adoption in the context of anomaly detection application. The document describes the architecture and design of anomaly detection algorithms developed on IBM InfoSphere Streams. Although the solution was designed to be used for cyber security, the implemented algorithms are agnostic regarding the data type that they monitor and therefore can detect anomalies in data from various industries such as healthcare, finance and retail. The document describes the implementation of two anomaly detection algorithms: KOAD and PCA. The KOAD algorithm performs online anomaly detection with incremental learning and the PCA algorithm in performs offline anomaly detection. The solution was designed to provide near real-time insight into low latency on large data volume observation.
ISBN:1467316318
9781467316316
ISSN:2158-9178
DOI:10.1109/ISPA.2012.145