MespaConfig: Memory-Sparing Configuration Auto-Tuning for Co-Located In-Memory Cluster Computing Jobs

Distributed in-memory computing frameworks usually have lots of parameters (e.g., the buffer size of shuffle) to form a configuration for each execution. A well-tuned configuration can bring large improvements of performance. However, to improve resource utilization, jobs are often share the same cl...

Celý popis

Uložené v:
Podrobná bibliografia
Vydané v:IEEE transactions on services computing Ročník 15; číslo 5; s. 2883 - 2896
Hlavní autori: Zong, Zan, Wen, Lijie, Hu, Xuming, Han, Rui, Qian, Chen, Lin, Li
Médium: Journal Article
Jazyk:English
Vydavateľské údaje: Piscataway IEEE 01.09.2022
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Predmet:
Cluster computing
Clusters
co-locate
Computation
Computational modeling
Configuration tuning
Configurations
Distributed memory
in-memory computing
Load modeling
memory-sparing
Optimization
Parallel processing
Performance enhancement
performance optimization
Predictive models
Resource utilization
Tuning
ISSN:1939-1374, 2372-0204
On-line prístup:Získať plný text
Tagy: Pridať tag
Žiadne tagy, Buďte prvý, kto otaguje tento záznam!
Popis
Shrnutí:Distributed in-memory computing frameworks usually have lots of parameters (e.g., the buffer size of shuffle) to form a configuration for each execution. A well-tuned configuration can bring large improvements of performance. However, to improve resource utilization, jobs are often share the same cluster, which causes dynamic cluster load conditions. According to our observation, the variation of cluster load reduces effectiveness of configuration tuning. Besides, as a common problem of cluster computing jobs, overestimation of resources also occurs during configuration tuning. It is challenging to efficiently find the optimal configuration in a shared cluster with the consideration of memory-sparing. In this article, we introduce MespaConfig, a job-level configuration optimizer for distributed in-memory computing jobs. Advancements of MespaConfig over previous work are features including memory-sparing and load-sensitive. We evaluate MespaConfig by 6 typical Spark programs under different load conditions. The evaluation results show that MespaConfig improves the performance of six typical programs by up to 12× compared with default configurations. MespaConfig also achieves at most 41 percent reduction of configuration memory usage and reduces the optimization time overhead by 10.8× compared with the state-of-the-art approach.
Bibliografia:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:1939-1374
2372-0204
DOI:10.1109/TSC.2021.3063118