dSimpleGraph: A Novel Distributed Clustering Algorithm for Exploring Very Large Scale Unknown Data Sets

Some of the major challenges in current clustering applications include: some data sets are so huge that it is difficult to load the entire data sets into memory for clustering, the data sets are often distributed over different locations for various reasons, which makes it impossible to process the...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:2010 IEEE International Conference on Data Mining Workshops S. 162 - 169
Hauptverfasser: Li Lu, Yunhong Gu, Grossman, R
Format: Tagungsbericht
Sprache:Englisch
Veröffentlicht: IEEE 01.12.2010
Schlagworte:
Algorithm design and analysis
Clustering algorithms
Complexity theory
Data mining
Distributed clustering algorithm
Distributed databases
Equivalence relation
MapReduce
Shape
ISBN:9781424492442, 1424492440
ISSN:2375-9232
Online-Zugang:Volltext
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Beschreibung
Zusammenfassung:Some of the major challenges in current clustering applications include: some data sets are so huge that it is difficult to load the entire data sets into memory for clustering, the data sets are often distributed over different locations for various reasons, which makes it impossible to process them centrally, and when lacking prior knowledge of the unknown data sets, it is troublesome to choose the appropriate parameters to feed into existing clustering algorithms. Therefore, a distributed clustering algorithm without too many parameters becomes rather appealing. Although some distributed clustering algorithms have been proposed, it is still a challenge for them to solve all of these problems. In this paper, we propose and implement a novel micro-cluster based distributed clustering algorithm called dSimpleGraph. An equivalence relation on two micro-clusters is defined. Relying on the relation, dSimpleGraph can efficiently cluster data on the local machines, moreover, it can easily generate a determined global view from local views. Only two scalar parameters are needed and the generated clusters can be any shape. Its MapReduce-style structure allows it to be easily implemented on existing distributed computing platforms. Extensive experimental studies show that dSimpleGraph is very fast and very suitable for exploring very large scale unknown data sets.
ISBN:9781424492442
1424492440
ISSN:2375-9232
DOI:10.1109/ICDMW.2010.12