REMOLD: An Efficient Model-Based Clustering Algorithm for Large Datasets with Spark

Density-based clustering algorithms have the distinctive advantage of discovering arbitrarily shaped clusters, but they usually require a procedure to compute the distance between every pair of data points, and this procedure is prohibitive for large datasets since it has quadratic computation compl...

Celý popis

Uloženo v:
Podrobná bibliografie
Vydáno v:2017 IEEE 23rd International Conference on Parallel and Distributed Systems (ICPADS) s. 376 - 383
Hlavní autoři: Liang, Mingfei, Li, Qingyong, Geng, Yangli-ao, Wang, Jianzhu, Wei, Zhi
Médium: Konferenční příspěvek
Jazyk:angličtina
Vydáno: IEEE 01.12.2017
Témata:
Clustering algorithms
Computational modeling
density estimation
density-based clustering
distributed clustering
Estimation
Gaussian model
Kernel
Load modeling
Partitioning algorithms
Spark
Sparks
ISSN:1521-9097
On-line přístup:Získat plný text
Tagy: Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
Popis
Shrnutí:Density-based clustering algorithms have the distinctive advantage of discovering arbitrarily shaped clusters, but they usually require a procedure to compute the distance between every pair of data points, and this procedure is prohibitive for large datasets since it has quadratic computation complexity. In this paper, we propose a new distributed clustering algorithm, named REstore MOdel with Local Density estimation (REMOLD). Firstly, REMODL applies a balanced partitioning method to evenly divide an large dataset based on Local Sensitive Hashing (LSH). Then, it locally clusters each partition of the dataset, and uses a Gaussian model to represent each local cluster based on the observation that the density distribution of each local cluster shares similar shape with Gaussian distribution. Finally, these models are aggregated on a server where REMOLD restores global clusters based on these local Gaussian models. More specifically, model connection, which measures the density connectivity between two models, are defined to merge local models with an optimized procedure. In this aggregation, REMOLD requires low cost of network transmission for local Gaussian models, since the number of Gaussian models is often less than that of core objects for each partition. We evaluate REMOLD on three synthetic datasets and three real-world datasets on Spark, and the experiment results demonstrate that REMOLD is efficient and effective to find out clusters with complex shapes and it outperforms the established methods.
ISSN:1521-9097
DOI:10.1109/ICPADS.2017.00057