REMOLD: An Efficient Model-Based Clustering Algorithm for Large Datasets with Spark

Density-based clustering algorithms have the distinctive advantage of discovering arbitrarily shaped clusters, but they usually require a procedure to compute the distance between every pair of data points, and this procedure is prohibitive for large datasets since it has quadratic computation compl...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:2017 IEEE 23rd International Conference on Parallel and Distributed Systems (ICPADS) S. 376 - 383
Hauptverfasser: Liang, Mingfei, Li, Qingyong, Geng, Yangli-ao, Wang, Jianzhu, Wei, Zhi
Format: Tagungsbericht
Sprache:Englisch
Veröffentlicht: IEEE 01.12.2017
Schlagworte:
Clustering algorithms
Computational modeling
density estimation
density-based clustering
distributed clustering
Estimation
Gaussian model
Kernel
Load modeling
Partitioning algorithms
Spark
Sparks
ISSN:1521-9097
Online-Zugang:Volltext
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Beschreibung
Zusammenfassung:Density-based clustering algorithms have the distinctive advantage of discovering arbitrarily shaped clusters, but they usually require a procedure to compute the distance between every pair of data points, and this procedure is prohibitive for large datasets since it has quadratic computation complexity. In this paper, we propose a new distributed clustering algorithm, named REstore MOdel with Local Density estimation (REMOLD). Firstly, REMODL applies a balanced partitioning method to evenly divide an large dataset based on Local Sensitive Hashing (LSH). Then, it locally clusters each partition of the dataset, and uses a Gaussian model to represent each local cluster based on the observation that the density distribution of each local cluster shares similar shape with Gaussian distribution. Finally, these models are aggregated on a server where REMOLD restores global clusters based on these local Gaussian models. More specifically, model connection, which measures the density connectivity between two models, are defined to merge local models with an optimized procedure. In this aggregation, REMOLD requires low cost of network transmission for local Gaussian models, since the number of Gaussian models is often less than that of core objects for each partition. We evaluate REMOLD on three synthetic datasets and three real-world datasets on Spark, and the experiment results demonstrate that REMOLD is efficient and effective to find out clusters with complex shapes and it outperforms the established methods.
ISSN:1521-9097
DOI:10.1109/ICPADS.2017.00057