Investigating the performance of Hadoop and Spark platforms on machine learning algorithms

One of the most challenging issues in the big data research area is the inability to process a large volume of information in a reasonable time. Hadoop and Spark are two frameworks for distributed data processing. Hadoop is a very popular and general platform for big data processing. Because of the...

Celý popis

Uložené v:
Podrobná bibliografia
Vydané v:The Journal of supercomputing Ročník 77; číslo 2; s. 1273 - 1300
Hlavní autori: Mostafaeipour, Ali, Jahangard Rafsanjani, Amir, Ahmadi, Mohammad, Arockia Dhanraj, Joshuva
Médium: Journal Article
Jazyk:English
Vydavateľské údaje: New York Springer US 01.02.2021
Springer Nature B.V
Predmet:
Algorithms
Big Data
Compilers
Computer Science
Data processing
Interpreters
Iterative algorithms
Iterative methods
K-nearest neighbors algorithm
Machine learning
Microprocessors
Processor Architectures
Programming Languages
Run time (computers)
Big data
Spark
Ganglia
Machine learning
Hadoop
MapReduce
ISSN:0920-8542, 1573-0484
On-line prístup:Získať plný text
Tagy: Pridať tag
Žiadne tagy, Buďte prvý, kto otaguje tento záznam!
Popis
Shrnutí:One of the most challenging issues in the big data research area is the inability to process a large volume of information in a reasonable time. Hadoop and Spark are two frameworks for distributed data processing. Hadoop is a very popular and general platform for big data processing. Because of the in-memory programming model, Spark as an open-source framework is suitable for processing iterative algorithms. In this paper, Hadoop and Spark frameworks, the big data processing platforms, are evaluated and compared in terms of runtime, memory and network usage, and central processor efficiency. Hence, the K-nearest neighbor (KNN) algorithm is implemented on datasets with different sizes within both Hadoop and Spark frameworks. The results show that the runtime of the KNN algorithm implemented on Spark is 4 to 4.5 times faster than Hadoop. Evaluations show that Hadoop uses more sources, including central processor and network. It is concluded that the CPU in Spark is more effective than Hadoop. On the other hand, the memory usage in Hadoop is less than Spark.
Bibliografia:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:0920-8542
1573-0484
DOI:10.1007/s11227-020-03328-5