Optimization Techniques for a Distributed In-Memory Computing Platform by Leveraging SSD

In this paper, we present several optimization strategies that can improve the overall performance of the distributed in-memory computing system, “Apache Spark”. Despite its distributed memory management capability for iterative jobs and intermediate data, Spark has a significant performance degrada...

Celý popis

Uložené v:
Podrobná bibliografia
Vydané v:Applied sciences Ročník 11; číslo 18; s. 8476
Hlavní autori: Choi, June, Lee, Jaehyun, Kim, Jik-Soo, Lee, Jaehwan
Médium: Journal Article
Jazyk:English
Vydavateľské údaje: Basel MDPI AG 01.09.2021
Predmet:
Apache Spark
Big Data
Data analysis
in-memory processing framework
Machine learning
memory management
Optimization techniques
PageRank
Sanitation services
solid-state drive
Workloads
ISSN:2076-3417, 2076-3417
On-line prístup:Získať plný text
Tagy: Pridať tag
Žiadne tagy, Buďte prvý, kto otaguje tento záznam!
Popis
Shrnutí:In this paper, we present several optimization strategies that can improve the overall performance of the distributed in-memory computing system, “Apache Spark”. Despite its distributed memory management capability for iterative jobs and intermediate data, Spark has a significant performance degradation problem when the available amount of main memory (DRAM, typically used for data caching) is limited. To address this problem, we leverage an SSD (solid-state drive) to supplement the lack of main memory bandwidth. Specifically, we present an effective optimization methodology for Apache Spark by collectively investigating the effects of changing the capacity fraction ratios of the shuffle and storage spaces in the “Spark JVM Heap Configuration” and applying different “RDD Caching Policies” (e.g., SSD-backed memory caching). Our extensive experimental results show that by utilizing the proposed optimization techniques, we can improve the overall performance by up to 42%.
Bibliografia:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:2076-3417
2076-3417
DOI:10.3390/app11188476