Optimization Techniques for a Distributed In-Memory Computing Platform by Leveraging SSD

In this paper, we present several optimization strategies that can improve the overall performance of the distributed in-memory computing system, “Apache Spark”. Despite its distributed memory management capability for iterative jobs and intermediate data, Spark has a significant performance degrada...

Full description

Saved in:
Bibliographic Details
Published in:Applied sciences Vol. 11; no. 18; p. 8476
Main Authors: Choi, June, Lee, Jaehyun, Kim, Jik-Soo, Lee, Jaehwan
Format: Journal Article
Language:English
Published: Basel MDPI AG 01.09.2021
Subjects:
Apache Spark
Big Data
Data analysis
in-memory processing framework
Machine learning
memory management
Optimization techniques
PageRank
Sanitation services
solid-state drive
Workloads
ISSN:2076-3417, 2076-3417
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:In this paper, we present several optimization strategies that can improve the overall performance of the distributed in-memory computing system, “Apache Spark”. Despite its distributed memory management capability for iterative jobs and intermediate data, Spark has a significant performance degradation problem when the available amount of main memory (DRAM, typically used for data caching) is limited. To address this problem, we leverage an SSD (solid-state drive) to supplement the lack of main memory bandwidth. Specifically, we present an effective optimization methodology for Apache Spark by collectively investigating the effects of changing the capacity fraction ratios of the shuffle and storage spaces in the “Spark JVM Heap Configuration” and applying different “RDD Caching Policies” (e.g., SSD-backed memory caching). Our extensive experimental results show that by utilizing the proposed optimization techniques, we can improve the overall performance by up to 42%.
Bibliography:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:2076-3417
2076-3417
DOI:10.3390/app11188476