dSimpleGraph: A Novel Distributed Clustering Algorithm for Exploring Very Large Scale Unknown Data Sets

Some of the major challenges in current clustering applications include: some data sets are so huge that it is difficult to load the entire data sets into memory for clustering, the data sets are often distributed over different locations for various reasons, which makes it impossible to process the...

Full description

Saved in:
Bibliographic Details
Published in:2010 IEEE International Conference on Data Mining Workshops pp. 162 - 169
Main Authors: Li Lu, Yunhong Gu, Grossman, R
Format: Conference Proceeding
Language:English
Published: IEEE 01.12.2010
Subjects:
Algorithm design and analysis
Clustering algorithms
Complexity theory
Data mining
Distributed clustering algorithm
Distributed databases
Equivalence relation
MapReduce
Shape
ISBN:9781424492442, 1424492440
ISSN:2375-9232
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Some of the major challenges in current clustering applications include: some data sets are so huge that it is difficult to load the entire data sets into memory for clustering, the data sets are often distributed over different locations for various reasons, which makes it impossible to process them centrally, and when lacking prior knowledge of the unknown data sets, it is troublesome to choose the appropriate parameters to feed into existing clustering algorithms. Therefore, a distributed clustering algorithm without too many parameters becomes rather appealing. Although some distributed clustering algorithms have been proposed, it is still a challenge for them to solve all of these problems. In this paper, we propose and implement a novel micro-cluster based distributed clustering algorithm called dSimpleGraph. An equivalence relation on two micro-clusters is defined. Relying on the relation, dSimpleGraph can efficiently cluster data on the local machines, moreover, it can easily generate a determined global view from local views. Only two scalar parameters are needed and the generated clusters can be any shape. Its MapReduce-style structure allows it to be easily implemented on existing distributed computing platforms. Extensive experimental studies show that dSimpleGraph is very fast and very suitable for exploring very large scale unknown data sets.
ISBN:9781424492442
1424492440
ISSN:2375-9232
DOI:10.1109/ICDMW.2010.12