Cost-effective and adaptive clustering algorithm for stream processing on cloud system

Clustering is a fundamental operation that plays an essential role in data management and analysis. Clustering algorithms have been well studied over the past two decades, but the real-time clustering has yet to be maturely applied. For applications based on clustering calculations, capturing the dy...

Full description

Saved in:
Bibliographic Details
Published in:GeoInformatica Vol. 27; no. 1; pp. 1 - 21
Main Authors: Xia, Yue, Fang, Junhua, Chao, Pingfu, Pan, Zhicheng, Shang, Jedi S.
Format: Journal Article
Language:English
Published: New York Springer US 01.01.2023
Springer
Springer Nature B.V
Subjects:
Adaptive algorithms
Algorithms
Clustering
Computer applications
Computer Science
Data management
Data Structures and Information Theory
Data transmission
Geographical Information Systems/Cartography
Information management
Information Storage and Retrieval
Management science
Multimedia Information Systems
Outliers (statistics)
Real time
Rivers
Workload
Real-time processing
Time interval
Window model
Density-based clustering
Cluster evolution
ISSN:1384-6175, 1573-7624
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Clustering is a fundamental operation that plays an essential role in data management and analysis. Clustering algorithms have been well studied over the past two decades, but the real-time clustering has yet to be maturely applied. For applications based on clustering calculations, capturing the dynamic changes of clusters and trends of moving objects in a real-time manner can maximize the value of the data. Although the DSPE ( D istributed S tream P rocessing E ngine) is capable of such workloads, it still faces the problems of fixed window size and computational resources waste. In this paper, we introduce a new C ost- e ffective and A daptive C lustering method ( CeAC ), which can improve computational efficiency while ensuring the accuracy of the clustering result. Specifically, we design a composite window model which contains the latest data records and maintains historical states. To achieve a lightweight clustering, we propose a fully online clustering algorithm based on grid density, which can capture clusters with arbitrary shape and effectively handle outliers in parallel. We further introduce an adaptive calculation model to accelerate the clustering operation by shedding workload according to the incoming data characteristic. Experimental results show that the proposed method is accurate and efficient in real-time data stream clustering.
Bibliography:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:1384-6175
1573-7624
DOI:10.1007/s10707-021-00442-1