A Performance Comparison of Big Data Processing Platform Based on Parallel Clustering Algorithms

The performance of three typical big data processing platform: Hadoop, Spark and DataMPI are compared based on different parallel clustering algorithms: parallel K-means, parallel fuzzy K-means and parallel Canopy. Experiments are performed on different text as well as numeric dataset and clusters o...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:Procedia computer science Jg. 139; S. 127 - 135
Hauptverfasser: Hai, Mo, Zhang, Yuejing, Li, Haifeng
Format: Journal Article
Sprache:Englisch
Veröffentlicht: Elsevier B.V 2018
Schlagworte:
Canopy
DataMPI
fuzzy K-means
Hadoop
K-means
Spark
K-means
DataMPI
Spark
fuzzy K-means
Hadoop
Canopy
ISSN:1877-0509, 1877-0509
Online-Zugang:Volltext
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Beschreibung
Zusammenfassung:The performance of three typical big data processing platform: Hadoop, Spark and DataMPI are compared based on different parallel clustering algorithms: parallel K-means, parallel fuzzy K-means and parallel Canopy. Experiments are performed on different text as well as numeric dataset and clusters of different scale. The results show that: (1) for the same data set, when the memory of each node is 4GB, DataMPI can achieve about 60% performance improvement compared with Hadoop, and can achieve about 32% performance improvement compared with Spark; (2) in order to obtain a high clustering performance, a cluster with 6 nodes and 6GB memory of each node should be selected.
ISSN:1877-0509
1877-0509
DOI:10.1016/j.procs.2018.10.228