MespaConfig: Memory-Sparing Configuration Auto-Tuning for Co-Located In-Memory Cluster Computing Jobs

Distributed in-memory computing frameworks usually have lots of parameters (e.g., the buffer size of shuffle) to form a configuration for each execution. A well-tuned configuration can bring large improvements of performance. However, to improve resource utilization, jobs are often share the same cl...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:IEEE transactions on services computing Jg. 15; H. 5; S. 2883 - 2896
Hauptverfasser: Zong, Zan, Wen, Lijie, Hu, Xuming, Han, Rui, Qian, Chen, Lin, Li
Format: Journal Article
Sprache:Englisch
Veröffentlicht: Piscataway IEEE 01.09.2022
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Schlagworte:
Cluster computing
Clusters
co-locate
Computation
Computational modeling
Configuration tuning
Configurations
Distributed memory
in-memory computing
Load modeling
memory-sparing
Optimization
Parallel processing
Performance enhancement
performance optimization
Predictive models
Resource utilization
Tuning
ISSN:1939-1374, 2372-0204
Online-Zugang:Volltext
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Beschreibung
Zusammenfassung:Distributed in-memory computing frameworks usually have lots of parameters (e.g., the buffer size of shuffle) to form a configuration for each execution. A well-tuned configuration can bring large improvements of performance. However, to improve resource utilization, jobs are often share the same cluster, which causes dynamic cluster load conditions. According to our observation, the variation of cluster load reduces effectiveness of configuration tuning. Besides, as a common problem of cluster computing jobs, overestimation of resources also occurs during configuration tuning. It is challenging to efficiently find the optimal configuration in a shared cluster with the consideration of memory-sparing. In this article, we introduce MespaConfig, a job-level configuration optimizer for distributed in-memory computing jobs. Advancements of MespaConfig over previous work are features including memory-sparing and load-sensitive. We evaluate MespaConfig by 6 typical Spark programs under different load conditions. The evaluation results show that MespaConfig improves the performance of six typical programs by up to 12× compared with default configurations. MespaConfig also achieves at most 41 percent reduction of configuration memory usage and reduces the optimization time overhead by 10.8× compared with the state-of-the-art approach.
Bibliographie:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:1939-1374
2372-0204
DOI:10.1109/TSC.2021.3063118