A Performance Comparison of Big Data Processing Platform Based on Parallel Clustering Algorithms

The performance of three typical big data processing platform: Hadoop, Spark and DataMPI are compared based on different parallel clustering algorithms: parallel K-means, parallel fuzzy K-means and parallel Canopy. Experiments are performed on different text as well as numeric dataset and clusters o...

Celý popis

Uloženo v:
Podrobná bibliografie
Vydáno v:Procedia computer science Ročník 139; s. 127 - 135
Hlavní autoři: Hai, Mo, Zhang, Yuejing, Li, Haifeng
Médium: Journal Article
Jazyk:angličtina
Vydáno: Elsevier B.V 2018
Témata:
Canopy
DataMPI
fuzzy K-means
Hadoop
K-means
Spark
K-means
DataMPI
Spark
fuzzy K-means
Hadoop
Canopy
ISSN:1877-0509, 1877-0509
On-line přístup:Získat plný text
Tagy: Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
Popis
Shrnutí:The performance of three typical big data processing platform: Hadoop, Spark and DataMPI are compared based on different parallel clustering algorithms: parallel K-means, parallel fuzzy K-means and parallel Canopy. Experiments are performed on different text as well as numeric dataset and clusters of different scale. The results show that: (1) for the same data set, when the memory of each node is 4GB, DataMPI can achieve about 60% performance improvement compared with Hadoop, and can achieve about 32% performance improvement compared with Spark; (2) in order to obtain a high clustering performance, a cluster with 6 nodes and 6GB memory of each node should be selected.
ISSN:1877-0509
1877-0509
DOI:10.1016/j.procs.2018.10.228