MespaConfig: Memory-Sparing Configuration Auto-Tuning for Co-Located In-Memory Cluster Computing Jobs

Distributed in-memory computing frameworks usually have lots of parameters (e.g., the buffer size of shuffle) to form a configuration for each execution. A well-tuned configuration can bring large improvements of performance. However, to improve resource utilization, jobs are often share the same cl...

Celý popis

Uložené v:

Podrobná bibliografia
Vydané v:	IEEE transactions on services computing Ročník 15; číslo 5; s. 2883 - 2896
Hlavní autori:	Zong, Zan, Wen, Lijie, Hu, Xuming, Han, Rui, Qian, Chen, Lin, Li
Médium:	Journal Article
Jazyk:	English
Vydavateľské údaje:	Piscataway IEEE 01.09.2022 The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Predmet:	Cluster computing Clusters co-locate Computation Computational modeling Configuration tuning Configurations Distributed memory in-memory computing Load modeling memory-sparing Optimization Parallel processing Performance enhancement performance optimization Predictive models Resource utilization Tuning
ISSN:	1939-1374, 2372-0204
On-line prístup:	Získať plný text
Tagy:	Pridať tag Žiadne tagy, Buďte prvý, kto otaguje tento záznam!

Abstract	Distributed in-memory computing frameworks usually have lots of parameters (e.g., the buffer size of shuffle) to form a configuration for each execution. A well-tuned configuration can bring large improvements of performance. However, to improve resource utilization, jobs are often share the same cluster, which causes dynamic cluster load conditions. According to our observation, the variation of cluster load reduces effectiveness of configuration tuning. Besides, as a common problem of cluster computing jobs, overestimation of resources also occurs during configuration tuning. It is challenging to efficiently find the optimal configuration in a shared cluster with the consideration of memory-sparing. In this article, we introduce MespaConfig, a job-level configuration optimizer for distributed in-memory computing jobs. Advancements of MespaConfig over previous work are features including memory-sparing and load-sensitive. We evaluate MespaConfig by 6 typical Spark programs under different load conditions. The evaluation results show that MespaConfig improves the performance of six typical programs by up to 12× compared with default configurations. MespaConfig also achieves at most 41 percent reduction of configuration memory usage and reduces the optimization time overhead by 10.8× compared with the state-of-the-art approach.
AbstractList	Distributed in-memory computing frameworks usually have lots of parameters (e.g., the buffer size of shuffle) to form a configuration for each execution. A well-tuned configuration can bring large improvements of performance. However, to improve resource utilization, jobs are often share the same cluster, which causes dynamic cluster load conditions. According to our observation, the variation of cluster load reduces effectiveness of configuration tuning. Besides, as a common problem of cluster computing jobs, overestimation of resources also occurs during configuration tuning. It is challenging to efficiently find the optimal configuration in a shared cluster with the consideration of memory-sparing. In this article, we introduce MespaConfig, a job-level configuration optimizer for distributed in-memory computing jobs. Advancements of MespaConfig over previous work are features including memory-sparing and load-sensitive. We evaluate MespaConfig by 6 typical Spark programs under different load conditions. The evaluation results show that MespaConfig improves the performance of six typical programs by up to 12× compared with default configurations. MespaConfig also achieves at most 41 percent reduction of configuration memory usage and reduces the optimization time overhead by 10.8× compared with the state-of-the-art approach.
Author	Wen, Lijie Qian, Chen Zong, Zan Hu, Xuming Lin, Li Han, Rui
Author_xml	– sequence: 1 givenname: Zan orcidid: 0000-0002-7828-9030 surname: Zong fullname: Zong, Zan email: zongz17@mails.tsinghua.edu.cn organization: School of Software, Tsinghua University, Beijing, China – sequence: 2 givenname: Lijie orcidid: 0000-0003-0358-3160 surname: Wen fullname: Wen, Lijie email: wenlj@tsinghua.edu.cn organization: School of Software, Tsinghua University, Beijing, China – sequence: 3 givenname: Xuming orcidid: 0000-0001-6075-4224 surname: Hu fullname: Hu, Xuming email: hxm19@mails.tsinghua.edu.cn organization: School of Software, Tsinghua University, Beijing, China – sequence: 4 givenname: Rui orcidid: 0000-0001-6894-1921 surname: Han fullname: Han, Rui email: hanrui@bit.edu.cn organization: School of Computer Science and Technology, Beijing Institute of Technology, Beijing, China – sequence: 5 givenname: Chen orcidid: 0000-0003-0155-8009 surname: Qian fullname: Qian, Chen email: qc16@mails.tsinghua.edu.cn organization: School of Software, Tsinghua University, Beijing, China – sequence: 6 givenname: Li surname: Lin fullname: Lin, Li email: lin-l16@mails.tsinghua.edu.cn organization: School of Software, Tsinghua University, Beijing, China
BookMark	eNp9kDtPwzAURi1UJNrCjsQSidnFj9Sx2aqIR1ErhpY5Ms51laqNg-0M_fckpGJgYLJ0v3Oudb8JGtWuBoRuKZlRStTDdpPPGGF0xonglMoLNGY8Y5gwko7QmCquMOVZeoUmIewJEUxKNUawhtDo3NW22j0mazg6f8KbRvuq3iXDuPU6Vq5OFm10eNvWfWKd71K8ckZHKJNljQc1yQ9tiNCHx6aNPfrmPsM1urT6EODm_E7Rx_PTNn_Fq_eXZb5YYcMUjVhnmqRWUWPKFBinQoKUtgQlmYCSl9ZCSebGWGEEBQG2A3Sacg1gpGaWT9H9sLfx7quFEIu9a33dfVmwjLF5KjIlOkoMlPEuBA-2MFX8uTF6XR0KSoq-0qKrtOgrLc6VdiL5Iza-Omp_-k-5G5QKAH5xxUVGSMq_AR_xhQo
CODEN	ITSCAD
CitedBy_id	crossref_primary_10_1186_s13677_023_00465_z crossref_primary_10_1145_3711119 crossref_primary_10_1109_TSC_2023_3325302 crossref_primary_10_1109_TPDS_2023_3266110 crossref_primary_10_1109_TCC_2024_3437484 crossref_primary_10_1007_s10586_024_04478_4 crossref_primary_10_3390_mi14030651
Cites_doi	10.1007/978-3-540-31880-4_11 10.1109/NAS.2017.8026871 10.1145/2523616.2523633 10.1109/ICDCS.2019.00012 10.1109/CLUSTER.2012.35 10.1016/B978-008045405-4.00149-X 10.1007/978-3-319-43283-0_2 10.1109/BigData.2017.8258257 10.1145/2371536.2371547 10.1145/3173162.3173187 10.1080/00031305.2016.1154108 10.1109/MSST.2010.5496972 10.1109/CLOUD.2016.0034 10.1145/2076022.1993495 10.1109/HPCC-SmartCity-DSS.2016.0053 10.1109/HPCC-SmartCity-DSS.2016.0088 10.7551/mitpress/3927.001.0001 10.1016/j.future.2017.08.011 10.1145/2600212.2600229 10.1145/3127479.3127492 10.1109/TPDS.2015.2449299 10.1023/A:1007618119488 10.1109/TPDS.2019.2923197 10.1145/781027.781052 10.1109/MASCOTS.2013.9 10.1145/2541940.2541941 10.1145/2391229.2391236 10.1145/3127479.3128605 10.1145/3267809.3267830 10.1214/19-AOS1931 10.5120/ijca2017913370 10.1109/4235.996017 10.1145/1327452.1327492
ContentType	Journal Article
Copyright	Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2022
Copyright_xml	– notice: Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2022
DBID	97E RIA RIE AAYXX CITATION 7SC 8FD JQ2 L7M L~C L~D
DOI	10.1109/TSC.2021.3063118
DatabaseName	IEEE Xplore (IEEE) IEEE All-Society Periodicals Package (ASPP) 1998–Present IEEE Electronic Library (IEL) CrossRef Computer and Information Systems Abstracts Technology Research Database ProQuest Computer Science Collection Advanced Technologies Database with Aerospace Computer and Information Systems Abstracts Academic Computer and Information Systems Abstracts Professional
DatabaseTitle	CrossRef Computer and Information Systems Abstracts Technology Research Database Computer and Information Systems Abstracts – Academic Advanced Technologies Database with Aerospace ProQuest Computer Science Collection Computer and Information Systems Abstracts Professional
DatabaseTitleList	Computer and Information Systems Abstracts
Database_xml	– sequence: 1 dbid: RIE name: IEEE Electronic Library (IEL) url: https://ieeexplore.ieee.org/ sourceTypes: Publisher
DeliveryMethod	fulltext_linktorsrc
Discipline	Engineering
EISSN	2372-0204
EndPage	2896
ExternalDocumentID	10_1109_TSC_2021_3063118 9367004
Genre	orig-research
GrantInformation_xml	– fundername: Tsinghua BNRist – fundername: National Natural Science Foundation of China; National Nature Science Foundation of China grantid: 71690231 funderid: 10.13039/501100001809 – fundername: National Key Research and Development Program of China grantid: 2019YFB1704003
GroupedDBID	0R~ 29I 4.4 5VS 6IK 97E AAJGR AARMG AASAJ AAWTH ABAZT ABJNI ABQJQ ABVLG ACGFO ACIWK AENEX AETIX AGQYO AGSQL AHBIQ AKJIK AKQYR ALMA_UNASSIGNED_HOLDINGS ATWAV BEFXN BFFAM BGNUA BKEBE BPEOZ EBS EJD HZ~ IEDLZ IFIPE IPLJI JAVBF M43 O9- OCL P2P PQQKQ RIA RIE RNI RNS RZB AAYXX CITATION 7SC 8FD JQ2 L7M L~C L~D
ID	FETCH-LOGICAL-c291t-a7a04f91ccd4e23168e88fde9826ed3dffed05ccf6c61e6ef68ea443aeec8a2f3
IEDL.DBID	RIE
ISICitedReferencesCount	8
ISICitedReferencesURI	http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000865092800029&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
ISSN	1939-1374
IngestDate	Sun Nov 09 07:04:32 EST 2025 Tue Nov 18 22:31:27 EST 2025 Sat Nov 29 02:04:35 EST 2025 Wed Aug 27 02:29:10 EDT 2025
IsPeerReviewed	true
IsScholarly	true
Issue	5
Language	English
License	https://ieeexplore.ieee.org/Xplorehelp/downloads/license-information/IEEE.html https://doi.org/10.15223/policy-029 https://doi.org/10.15223/policy-037
LinkModel	DirectLink
MergedId	FETCHMERGED-LOGICAL-c291t-a7a04f91ccd4e23168e88fde9826ed3dffed05ccf6c61e6ef68ea443aeec8a2f3
Notes	ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 14
ORCID	0000-0002-7828-9030 0000-0001-6075-4224 0000-0003-0155-8009 0000-0003-0358-3160 0000-0001-6894-1921
PQID	2722546796
PQPubID	85503
PageCount	14
ParticipantIDs	proquest_journals_2722546796 crossref_citationtrail_10_1109_TSC_2021_3063118 ieee_primary_9367004 crossref_primary_10_1109_TSC_2021_3063118
PublicationCentury	2000
PublicationDate	2022-09-01
PublicationDateYYYYMMDD	2022-09-01
PublicationDate_xml	– month: 09 year: 2022 text: 2022-09-01 day: 01
PublicationDecade	2020
PublicationPlace	Piscataway
PublicationPlace_xml	– name: Piscataway
PublicationTitle	IEEE transactions on services computing
PublicationTitleAbbrev	TSC
PublicationYear	2022
Publisher	IEEE The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher_xml	– name: IEEE – name: The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
References	ref13 ref35 ref12 ref15 ref37 ref14 ref36 ref52 ref11 ref10 ref32 ref17 ref38 ref19 ref18 Zaharia (ref1) ref50 ref24 ref46 ref23 ref45 ref26 ref48 ref25 ref47 ref42 ref41 ref21 ref49 ref29 Seber (ref33) 2012; 329 Boutin (ref51) ref8 ref7 Ousterhout (ref28) Lu (ref31) ref4 ref3 ref6 Iorgulescu (ref22) ref5 Haykin (ref30) 2009
References_xml	– ident: ref37 doi: 10.1007/978-3-540-31880-4_11 – ident: ref52 doi: 10.1109/NAS.2017.8026871 – ident: ref29 doi: 10.1145/2523616.2523633 – ident: ref14 doi: 10.1109/ICDCS.2019.00012 – ident: ref10 doi: 10.1109/CLUSTER.2012.35 – ident: ref26 doi: 10.1016/B978-008045405-4.00149-X – ident: ref19 doi: 10.1007/978-3-319-43283-0_2 – ident: ref25 article-title: A python package for statistical computations – ident: ref21 doi: 10.1109/BigData.2017.8258257 – ident: ref4 doi: 10.1145/2371536.2371547 – ident: ref3 doi: 10.1145/3173162.3173187 – ident: ref24 doi: 10.1080/00031305.2016.1154108 – ident: ref38 doi: 10.1109/MSST.2010.5496972 – ident: ref45 doi: 10.1109/CLOUD.2016.0034 – ident: ref48 doi: 10.1145/2076022.1993495 – ident: ref49 doi: 10.1109/HPCC-SmartCity-DSS.2016.0053 – ident: ref5 doi: 10.1109/HPCC-SmartCity-DSS.2016.0088 – ident: ref42 doi: 10.7551/mitpress/3927.001.0001 – ident: ref8 doi: 10.1016/j.future.2017.08.011 – ident: ref17 doi: 10.1145/2600212.2600229 – ident: ref23 doi: 10.1145/3127479.3127492 – ident: ref6 doi: 10.1109/TPDS.2015.2449299 – ident: ref15 article-title: Apache spark survey from typesafe – ident: ref18 doi: 10.1023/A:1007618119488 – ident: ref50 doi: 10.1109/TPDS.2019.2923197 – volume-title: Neural Networks and Learning Machines year: 2009 ident: ref30 – ident: ref41 doi: 10.1145/781027.781052 – ident: ref47 doi: 10.1109/MASCOTS.2013.9 – volume: 329 volume-title: Linear Regression Analysis year: 2012 ident: ref33 – start-page: 285 volume-title: Proc. USENIX 11th USENIX Conf. Operating Syst. Des. Implement. ident: ref51 article-title: Apollo: Scalable and coordinated scheduling for cloud-scale computing – start-page: 293 volume-title: Proc. 12th USENIX Conf. Netw. Syst. Des. Implementation ident: ref28 article-title: Making sense of performance in data analytics frameworks – ident: ref13 doi: 10.1145/2541940.2541941 – ident: ref12 doi: 10.1145/2391229.2391236 – start-page: 97 volume-title: Proc. USENIX Conf. Usenix Annu. Tech. Conf. ident: ref22 article-title: Don’t cry over spilled records: Memory elasticity of data-parallel applications and its application to cluster scheduling – ident: ref7 doi: 10.1145/3127479.3128605 – ident: ref11 doi: 10.1145/3267809.3267830 – start-page: 6232 volume-title: Proc. 31st Int. Conf. Neural Inf. Process. Syst. ident: ref31 article-title: The expressive power of neural networks: A view from the width – ident: ref32 doi: 10.1214/19-AOS1931 – ident: ref36 doi: 10.5120/ijca2017913370 – ident: ref35 doi: 10.1109/4235.996017 – start-page: 95 volume-title: Proc. 2nd USENIX Conf. Hot Top. Cloud Comput. ident: ref1 article-title: Spark: Cluster computing with working sets – ident: ref46 doi: 10.1145/1327452.1327492
SSID	ssj0062889
Score	2.3477416
Snippet	Distributed in-memory computing frameworks usually have lots of parameters (e.g., the buffer size of shuffle) to form a configuration for each execution. A...
SourceID	proquest crossref ieee
SourceType	Aggregation Database Enrichment Source Index Database Publisher
StartPage	2883
SubjectTerms	Cluster computing Clusters co-locate Computation Computational modeling Configuration tuning Configurations Distributed memory in-memory computing Load modeling memory-sparing Optimization Parallel processing Performance enhancement performance optimization Predictive models Resource utilization Tuning
Title	MespaConfig: Memory-Sparing Configuration Auto-Tuning for Co-Located In-Memory Cluster Computing Jobs
URI	https://ieeexplore.ieee.org/document/9367004 https://www.proquest.com/docview/2722546796
Volume	15
WOSCitedRecordID	wos000865092800029&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
hasFullText	1
inHoldings	1
isFullTextHit
isPrint
journalDatabaseRights	– providerCode: PRVIEE databaseName: IEEE Electronic Library (IEL) customDbUrl: eissn: 2372-0204 dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0062889 issn: 1939-1374 databaseCode: RIE dateStart: 20080101 isFulltext: true titleUrlDefault: https://ieeexplore.ieee.org/ providerName: IEEE
link	http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV3fS8MwEA46fNAHf4vTKXnwRTBbf6RN49sYDhU3BCf4VtLkMgbSilsF_3tzbTcERfCttHel9Gsvl-Tu-wi5EGEkMpGFLE54xjgELg4qzpkFZROjwEJSNQo_iPE4eXmRj2vkatULAwBV8Rl08bDayzeFLnGprCeRbQzJP9eFiOterWXURdVcudyG9GRv8jRwk7_A77qUOPRR0uPbsFPpqPwIvtWIMtz537Psku0mc6T9Guo9sgb5Ptn6xid4QGAELkBgE99sek1HWET7yZ5QZzCf0vp0WSNO--WiYJMSV0Woy1vdVfZQYHWUoXc5q13p4LVEHgVaSz-g6X2RzQ_J8_BmMrhljY4C04H0F0wJ5XErfa2NAwOVqiBJrAHpphZgQmMtGC_S2sY69iEG6wwcXqEC0IkKbHhEWnmRwzGhJvEUV0ZEUSx4JCN369AEPHKZRpZx32uT3vI1p7ohGUeti9e0mmx4MnXApAhM2gDTJpcrj7eaYOMP2wMEYmXXYNAmnSWSafMHztNABEj1L2R88rvXKdkMsJWhqhfrkNbivYQzsqE_FrP5-3n1cX0B0ufQCw
linkProvider	IEEE
linkToHtml	http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1Ra9swED5KV9j6sG7ryrK1mx72MpgaW5Ytq28lNKRdEgbNoG9Glk6lUOzRxIX---psJxQ6Cnsz9p0x_uzTSbr7PoDvKklVqcqEZ7ksuUQR4qCRkns0PncGPeZto_BUzef51ZX-vQU_N70wiNgWn-ExHbZ7-a62DS2VDTWxjRH556tUShF13VrruEu6uXq9ERnp4eJyFKZ_Ij4OSXESk6jHk4GnVVJ5Fn7bMWW8939P8w7e9rkjO-3Afg9bWH2A3SeMgvuAMwwhgtr4bq5P2IzKaB_4JSkNVtesO910mLPTZlXzRUPrIixkruEqn9ZUH-XYecU7Vza6bYhJgXXiD2R6UZfLj_BnfLYYTXivpMCt0PGKG2Ui6XVsrQtwkFYV5rl3qMPkAl3ivEcXpdb6zGYxZuiDQUAsMYg2N8InB7Bd1RV-AubyyEjjVJpmSqY6DbdOnJBpyDXKUsbRAIbr11zYnmac1C5ui3a6EekiAFMQMEUPzAB-bDz-dhQbL9juExAbux6DARyukSz6f3BZCCWI7F_p7PO_vb7B68liNi2m5_NfX-CNoMaGtnrsELZXdw0ewY69X90s7762H9ojyzvTUg
openUrl	ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=MespaConfig%3A+Memory-Sparing+Configuration+Auto-Tuning+for+Co-Located+In-Memory+Cluster+Computing+Jobs&rft.jtitle=IEEE+transactions+on+services+computing&rft.au=Zong%2C+Zan&rft.au=Wen%2C+Lijie&rft.au=Hu%2C+Xuming&rft.au=Han%2C+Rui&rft.date=2022-09-01&rft.pub=IEEE&rft.eissn=2372-0204&rft.volume=15&rft.issue=5&rft.spage=2883&rft.epage=2896&rft_id=info:doi/10.1109%2FTSC.2021.3063118&rft.externalDocID=9367004
thumbnail_l	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=1939-1374&client=summon
thumbnail_m	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=1939-1374&client=summon
thumbnail_s	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=1939-1374&client=summon