A Performance Comparison of Big Data Processing Platform Based on Parallel Clustering Algorithms

The performance of three typical big data processing platform: Hadoop, Spark and DataMPI are compared based on different parallel clustering algorithms: parallel K-means, parallel fuzzy K-means and parallel Canopy. Experiments are performed on different text as well as numeric dataset and clusters o...

Full description

Saved in:
Bibliographic Details
Published in:Procedia computer science Vol. 139; pp. 127 - 135
Main Authors: Hai, Mo, Zhang, Yuejing, Li, Haifeng
Format: Journal Article
Language:English
Published: Elsevier B.V 2018
Subjects:
Canopy
DataMPI
fuzzy K-means
Hadoop
K-means
Spark
K-means
DataMPI
Spark
fuzzy K-means
Hadoop
Canopy
ISSN:1877-0509, 1877-0509
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:The performance of three typical big data processing platform: Hadoop, Spark and DataMPI are compared based on different parallel clustering algorithms: parallel K-means, parallel fuzzy K-means and parallel Canopy. Experiments are performed on different text as well as numeric dataset and clusters of different scale. The results show that: (1) for the same data set, when the memory of each node is 4GB, DataMPI can achieve about 60% performance improvement compared with Hadoop, and can achieve about 32% performance improvement compared with Spark; (2) in order to obtain a high clustering performance, a cluster with 6 nodes and 6GB memory of each node should be selected.
ISSN:1877-0509
1877-0509
DOI:10.1016/j.procs.2018.10.228