Asymptotically Optimal Coded Distributed Computing via Combinatorial Designs
Coded distributed computing (CDC) introduced by Li et al. can greatly reduce the communication load for MapReduce computing systems. In the cascaded CDC with <inline-formula> <tex-math notation="LaTeX">K </tex-math></inline-formula> workers, <inline-formula> &...
Uložené v:
| Vydané v: | IEEE/ACM transactions on networking Ročník 32; číslo 4; s. 3018 - 3033 |
|---|---|
| Hlavní autori: | , , , |
| Médium: | Journal Article |
| Jazyk: | English |
| Vydavateľské údaje: |
IEEE
01.08.2024
|
| Predmet: | |
| ISSN: | 1063-6692, 1558-2566 |
| On-line prístup: | Získať plný text |
| Tagy: |
Pridať tag
Žiadne tagy, Buďte prvý, kto otaguje tento záznam!
|
| Abstract | Coded distributed computing (CDC) introduced by Li et al. can greatly reduce the communication load for MapReduce computing systems. In the cascaded CDC with <inline-formula> <tex-math notation="LaTeX">K </tex-math></inline-formula> workers, <inline-formula> <tex-math notation="LaTeX">N </tex-math></inline-formula> input files and <inline-formula> <tex-math notation="LaTeX">Q </tex-math></inline-formula> output functions, each input file will be mapped by <inline-formula> <tex-math notation="LaTeX">r </tex-math></inline-formula> workers and each output function will be computed by <inline-formula> <tex-math notation="LaTeX">s </tex-math></inline-formula> workers such that coding techniques can be applied to create multicast opportunities. The main drawback of most existing CDC schemes is that they require the original data to be split into a large number of input files that grows exponentially with <inline-formula> <tex-math notation="LaTeX">K </tex-math></inline-formula>, which would significantly increase the coding complexity and degrade the system performance. In this paper, we first use a classical combinatorial structure <inline-formula> <tex-math notation="LaTeX">t </tex-math></inline-formula>-design, for any integer <inline-formula> <tex-math notation="LaTeX">t\geq 2 </tex-math></inline-formula>, to develop a low-complexity and communication-efficient CDC with <inline-formula> <tex-math notation="LaTeX">r=s </tex-math></inline-formula>. Our scheme has much smaller <inline-formula> <tex-math notation="LaTeX">N </tex-math></inline-formula> and <inline-formula> <tex-math notation="LaTeX">Q </tex-math></inline-formula> than the existing schemes under the same parameters <inline-formula> <tex-math notation="LaTeX">K </tex-math></inline-formula>, <inline-formula> <tex-math notation="LaTeX">r </tex-math></inline-formula>, and <inline-formula> <tex-math notation="LaTeX">s </tex-math></inline-formula>; and achieves smaller communication loads compared with the state-of-the-art schemes when <inline-formula> <tex-math notation="LaTeX">K </tex-math></inline-formula> is relatively large. Remarkably, unlike the previous schemes that realize on large operation fields, our scheme operates in one-shot communication on the minimum binary field <inline-formula> <tex-math notation="LaTeX">\mathbb {F}_{2} </tex-math></inline-formula>. With a derived lower bound on the communication load under one-shot linear delivery, we show that the <inline-formula> <tex-math notation="LaTeX">t </tex-math></inline-formula>-design scheme is asymptotically optimal. Furthermore, we show that our construction method can incorporate the other combinatorial structures that have a similar property to <inline-formula> <tex-math notation="LaTeX">t </tex-math></inline-formula>-design. For instance, we use <inline-formula> <tex-math notation="LaTeX">t </tex-math></inline-formula>-GDD to obtain another one-shot asymptotically optimal CDC scheme over <inline-formula> <tex-math notation="LaTeX">\mathbb {F}_{2} </tex-math></inline-formula> that has different parameters from <inline-formula> <tex-math notation="LaTeX">t </tex-math></inline-formula>-design. Finally, we show that our construction method can also be used to construct CDC schemes with <inline-formula> <tex-math notation="LaTeX">r\neq s </tex-math></inline-formula> that have small file number and output function number. |
|---|---|
| AbstractList | Coded distributed computing (CDC) introduced by Li et al. can greatly reduce the communication load for MapReduce computing systems. In the cascaded CDC with <inline-formula> <tex-math notation="LaTeX">K </tex-math></inline-formula> workers, <inline-formula> <tex-math notation="LaTeX">N </tex-math></inline-formula> input files and <inline-formula> <tex-math notation="LaTeX">Q </tex-math></inline-formula> output functions, each input file will be mapped by <inline-formula> <tex-math notation="LaTeX">r </tex-math></inline-formula> workers and each output function will be computed by <inline-formula> <tex-math notation="LaTeX">s </tex-math></inline-formula> workers such that coding techniques can be applied to create multicast opportunities. The main drawback of most existing CDC schemes is that they require the original data to be split into a large number of input files that grows exponentially with <inline-formula> <tex-math notation="LaTeX">K </tex-math></inline-formula>, which would significantly increase the coding complexity and degrade the system performance. In this paper, we first use a classical combinatorial structure <inline-formula> <tex-math notation="LaTeX">t </tex-math></inline-formula>-design, for any integer <inline-formula> <tex-math notation="LaTeX">t\geq 2 </tex-math></inline-formula>, to develop a low-complexity and communication-efficient CDC with <inline-formula> <tex-math notation="LaTeX">r=s </tex-math></inline-formula>. Our scheme has much smaller <inline-formula> <tex-math notation="LaTeX">N </tex-math></inline-formula> and <inline-formula> <tex-math notation="LaTeX">Q </tex-math></inline-formula> than the existing schemes under the same parameters <inline-formula> <tex-math notation="LaTeX">K </tex-math></inline-formula>, <inline-formula> <tex-math notation="LaTeX">r </tex-math></inline-formula>, and <inline-formula> <tex-math notation="LaTeX">s </tex-math></inline-formula>; and achieves smaller communication loads compared with the state-of-the-art schemes when <inline-formula> <tex-math notation="LaTeX">K </tex-math></inline-formula> is relatively large. Remarkably, unlike the previous schemes that realize on large operation fields, our scheme operates in one-shot communication on the minimum binary field <inline-formula> <tex-math notation="LaTeX">\mathbb {F}_{2} </tex-math></inline-formula>. With a derived lower bound on the communication load under one-shot linear delivery, we show that the <inline-formula> <tex-math notation="LaTeX">t </tex-math></inline-formula>-design scheme is asymptotically optimal. Furthermore, we show that our construction method can incorporate the other combinatorial structures that have a similar property to <inline-formula> <tex-math notation="LaTeX">t </tex-math></inline-formula>-design. For instance, we use <inline-formula> <tex-math notation="LaTeX">t </tex-math></inline-formula>-GDD to obtain another one-shot asymptotically optimal CDC scheme over <inline-formula> <tex-math notation="LaTeX">\mathbb {F}_{2} </tex-math></inline-formula> that has different parameters from <inline-formula> <tex-math notation="LaTeX">t </tex-math></inline-formula>-design. Finally, we show that our construction method can also be used to construct CDC schemes with <inline-formula> <tex-math notation="LaTeX">r\neq s </tex-math></inline-formula> that have small file number and output function number. |
| Author | Cheng, Minquan Wu, Dianhua Wu, Youlong Li, Xianxian |
| Author_xml | – sequence: 1 givenname: Minquan orcidid: 0000-0003-0360-0610 surname: Cheng fullname: Cheng, Minquan email: chengqinshi@hotmail.com organization: Key Laboratory of Education Blockchain and Intelligent Technology, Ministry of Education, and the Guangxi Key Laboratory of Multi-Source Information Mining and Security, Guangxi Normal University, Guilin, China – sequence: 2 givenname: Youlong orcidid: 0000-0002-4383-9995 surname: Wu fullname: Wu, Youlong email: wuyl1@shanghaitech.edu.cn organization: School of Information Science and Technology, ShanghaiTech University, Shanghai, China – sequence: 3 givenname: Xianxian orcidid: 0000-0002-7083-3847 surname: Li fullname: Li, Xianxian email: lixx@gxnu.edu.cn organization: Key Laboratory of Education Blockchain and Intelligent Technology, Ministry of Education, and the Guangxi Key Laboratory of Multi-Source Information Mining and Security, Guangxi Normal University, Guilin, China – sequence: 4 givenname: Dianhua orcidid: 0000-0002-2966-0606 surname: Wu fullname: Wu, Dianhua email: dhwu@gxnu.edu.cn organization: School of Mathematics and Statistics, Guangxi Normal University, Guilin, China |
| BookMark | eNp9kMlqwzAQQEVJoUnaDyj04B-wqyVafAxOukBoLunZSJYUVGzLSEohf1-b5FB66GlmYN4sbwFmve8NAI8IFgjB8vnwsT0UGOJVQQjHrBQ3YI4oFTmmjM3GHDKSM1biO7CI8QtCRCBmc7Bbx3M3JJ9cI9v2nO2H5DrZZpXXRmcbF1Nw6pTGvPLdcEquP2bfTk6Vcr1MPrixe2OiO_bxHtxa2UbzcI1L8PmyPVRv-W7_-l6td3mDGUu5hoIIIqEiJdIriUhpG4qVodpCoyynGgstmdDaWC6oIlArbqnmnDHVIE2WAF3mNsHHGIythzBeHc41gvWko5501JOO-qpjZPgfpnFJJuf7FKRr_yWfLqQzxvzatGK4HP_4ARphca4 |
| CODEN | IEANEP |
| CitedBy_id | crossref_primary_10_1109_TMC_2025_3570907 crossref_primary_10_1016_j_comnet_2025_111381 |
| Cites_doi | 10.1109/TCOMM.2021.3087628 10.1109/TIT.2017.2725272 10.1109/ACCESS.2020.3043825 10.1109/TIT.2017.2756959 10.1090/memo/1406 10.1201/9781003040897 10.1109/TCOMM.2021.3087788 10.1109/TCOMM.2020.3030667 10.1109/TCOMM.2022.3211932 10.1109/TIT.2015.2504556 10.1109/TNSE.2021.3095040 10.1109/TIT.2020.2999675 10.1109/TCOMM.2021.3049821 10.1109/TPDS.2016.2587645 10.1145/2248487.2150984 10.1109/TIT.2019.2904055 10.5555/1863103.1863113 10.1145/2043164.2018448 10.1109/COMST.2021.3091684 10.1145/1327452.1327492 10.1109/ICC.2017.7996730 10.1109/ITW46852.2021.9457580 10.1007/b97564 10.1109/ISIT44484.2020.9174132 |
| ContentType | Journal Article |
| DBID | 97E RIA RIE AAYXX CITATION |
| DOI | 10.1109/TNET.2024.3372698 |
| DatabaseName | IEEE All-Society Periodicals Package (ASPP) 2005–Present IEEE All-Society Periodicals Package (ASPP) 1998–Present IEEE/IET Electronic Library CrossRef |
| DatabaseTitle | CrossRef |
| DatabaseTitleList | |
| Database_xml | – sequence: 1 dbid: RIE name: IEEE/IET Electronic Library url: https://ieeexplore.ieee.org/ sourceTypes: Publisher |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Engineering |
| EISSN | 1558-2566 |
| EndPage | 3033 |
| ExternalDocumentID | 10_1109_TNET_2024_3372698 10462938 |
| Genre | orig-research |
| GrantInformation_xml | – fundername: NSFC grantid: 62061004; U21A20474; 12161010; 2022GXNSFDA035087 funderid: 10.13039/501100001809 – fundername: Guangxi Collaborative Innovation Center of Multi-Source Information Integration and Intelligent Processing – fundername: Guangxi Bagui Scholar Teams for Innovation and Research Project |
| GroupedDBID | -DZ -~X .DC 0R~ 29I 4.4 5GY 5VS 6IK 85S 8US 97E 9M8 AAJGR AAKMM AALFJ AARMG AASAJ AAWTH AAWTV ABAZT ABPPZ ABQJQ ABVLG ACGFS ACGOD ACIWK ACM ADBCU ADL AEBYY AEFXT AEJOY AENSD AETEA AETIX AFWIH AFWXC AGQYO AGSQL AHBIQ AI. AIBXA AIKLT AKJIK AKQYR AKRVB ALLEH ALMA_UNASSIGNED_HOLDINGS ATWAV BDXCO BEFXN BFFAM BGNUA BKEBE BPEOZ CCLIF CS3 D0L EBS EJD FEDTE GUFHI HF~ HGAVV HZ~ H~9 I07 ICLAB IEDLZ IES IFIPE IFJZH IPLJI JAVBF LAI LHSKQ M43 MVM O9- OCL P1C P2P PQQKQ RIA RIE RNS ROL TN5 UPT UQL VH1 XOL YR2 ZCA AAYXX CITATION |
| ID | FETCH-LOGICAL-c266t-d08383a0b391d4a139fc52be5df0ebf75d28da68ddef785b30db7f5d7766bc1d3 |
| IEDL.DBID | RIE |
| ISICitedReferencesCount | 4 |
| ISICitedReferencesURI | http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=001185949900001&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| ISSN | 1063-6692 |
| IngestDate | Tue Nov 18 21:44:58 EST 2025 Sat Nov 29 03:05:28 EST 2025 Wed Aug 27 01:54:24 EDT 2025 |
| IsPeerReviewed | true |
| IsScholarly | true |
| Issue | 4 |
| Language | English |
| License | https://ieeexplore.ieee.org/Xplorehelp/downloads/license-information/IEEE.html https://doi.org/10.15223/policy-029 https://doi.org/10.15223/policy-037 |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-c266t-d08383a0b391d4a139fc52be5df0ebf75d28da68ddef785b30db7f5d7766bc1d3 |
| ORCID | 0000-0003-0360-0610 0000-0002-4383-9995 0000-0002-2966-0606 0000-0002-7083-3847 |
| PageCount | 16 |
| ParticipantIDs | crossref_citationtrail_10_1109_TNET_2024_3372698 crossref_primary_10_1109_TNET_2024_3372698 ieee_primary_10462938 |
| PublicationCentury | 2000 |
| PublicationDate | 2024-08-01 |
| PublicationDateYYYYMMDD | 2024-08-01 |
| PublicationDate_xml | – month: 08 year: 2024 text: 2024-08-01 day: 01 |
| PublicationDecade | 2020 |
| PublicationTitle | IEEE/ACM transactions on networking |
| PublicationTitleAbbrev | TNET |
| PublicationYear | 2024 |
| Publisher | IEEE |
| Publisher_xml | – name: IEEE |
| References | ref13 ref12 ref15 ref14 ref11 ref10 ref2 ref1 ref17 ref19 ref18 ref24 ref23 Keevash (ref25) 2014 ref26 ref20 ref22 ref21 Chang (ref27) 1976 ref8 ref7 ref9 ref4 Wang (ref16) 2022 ref3 ref6 ref5 |
| References_xml | – year: 2022 ident: ref16 article-title: Coded distributed computing with pre-set assignments of data and output functions publication-title: arXiv:2201.06300 – ident: ref11 doi: 10.1109/TCOMM.2021.3087628 – ident: ref20 doi: 10.1109/TIT.2017.2725272 – ident: ref19 doi: 10.1109/ACCESS.2020.3043825 – ident: ref4 doi: 10.1109/TIT.2017.2756959 – ident: ref26 doi: 10.1090/memo/1406 – ident: ref23 doi: 10.1201/9781003040897 – ident: ref21 doi: 10.1109/TCOMM.2021.3087788 – ident: ref15 doi: 10.1109/TCOMM.2020.3030667 – ident: ref22 doi: 10.1109/TCOMM.2022.3211932 – year: 1976 ident: ref27 article-title: An existence theory for group divisible designs – ident: ref18 doi: 10.1109/TIT.2015.2504556 – ident: ref13 doi: 10.1109/TNSE.2021.3095040 – ident: ref8 doi: 10.1109/TIT.2020.2999675 – ident: ref12 doi: 10.1109/TCOMM.2021.3049821 – ident: ref1 doi: 10.1109/TPDS.2016.2587645 – ident: ref2 doi: 10.1145/2248487.2150984 – ident: ref14 doi: 10.1109/TIT.2019.2904055 – ident: ref6 doi: 10.5555/1863103.1863113 – ident: ref3 doi: 10.1145/2043164.2018448 – ident: ref17 doi: 10.1109/COMST.2021.3091684 – ident: ref5 doi: 10.1145/1327452.1327492 – year: 2014 ident: ref25 article-title: The existence of designs publication-title: arXiv:1401.3665 – ident: ref9 doi: 10.1109/ICC.2017.7996730 – ident: ref10 doi: 10.1109/ITW46852.2021.9457580 – ident: ref24 doi: 10.1007/b97564 – ident: ref7 doi: 10.1109/ISIT44484.2020.9174132 |
| SSID | ssj0013026 |
| Score | 2.4677463 |
| Snippet | Coded distributed computing (CDC) introduced by Li et al. can greatly reduce the communication load for MapReduce computing systems. In the cascaded CDC with... |
| SourceID | crossref ieee |
| SourceType | Enrichment Source Index Database Publisher |
| StartPage | 3018 |
| SubjectTerms | asymptotically optimal Coded distributed computing Costs Distributed computing Encoding Symbols t-design t-GDD Task analysis Technological innovation Vectors |
| Title | Asymptotically Optimal Coded Distributed Computing via Combinatorial Designs |
| URI | https://ieeexplore.ieee.org/document/10462938 |
| Volume | 32 |
| WOSCitedRecordID | wos001185949900001&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| journalDatabaseRights | – providerCode: PRVIEE databaseName: IEEE/IET Electronic Library customDbUrl: eissn: 1558-2566 dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0013026 issn: 1063-6692 databaseCode: RIE dateStart: 19930101 isFulltext: true titleUrlDefault: https://ieeexplore.ieee.org/ providerName: IEEE |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV07a8MwEBZN6NAOfaY0feGhU8GJLFuyNIY86FDSDilkM5IlQSAvEieQf9-T7Lbp0EI32ejAfJLvdLrHh9BjnkeuAISHVEQyTIhkobLchmALsMaJoaCXPdlEOhzy8Vi8VcXqvhbGGOOTz0zLDX0sXy_yjbsqa7t4JJgnXkO1NGVlsdZ3yAB7bjVwceKQMUGqEGaERXs07I_AFSRJK45TwgT_YYT2WFW8URmc_vNzztBJdXoMOuVyn6MDM79Ax3s9BS_RS2e9my2Lhb-jnu6CV9AJM5DpLrTRQc_1yXUUVzAuCR1AKNhOpHsCJ9m54LAjg55P7Fg30PugP-o-hxVlQpgDukWo4UTFY4lVLCKdSDje2ZwSZai22CibUk24loyDUrMppyrGWqWWaoCTqTzS8RWqzxdzc42C1EpBLBXEsfmqSCrXFocS-MG1FZKZJsKfGGZ51U_c0VpMM-9XYJE52DMHe1bB3kRPXyLLspnGX5MbDvK9iSXaN7-8v0VHTrxMzrtD9WK1MffoMN8Wk_Xqwe-VD-aDvOA |
| linkProvider | IEEE |
| linkToHtml | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV3NS8MwFA86BfXg58T52YMnoTNNmzY5jn0wcVYPFXYrSZPAYFvH1g3235ukVedBwVta8kr5NX0vL-_jB8B9lnmmAIS4mHrMDRALXa6IcrUtgAIGEmu9bMkmojgmwyF9q4rVbS2MlNImn8mmGdpYvsizpTkqezTxSG2eyDbYMdRZVbnWd9AAWnY17eT4bhhSVAUxPUgfk7ibaGcQBU3fj1BIyQ8ztMGrYs1K7-ifL3QMDqv9o9MqP_gJ2JLTU3Cw0VXwDAxai_VkVuT2lHq8dl61VphomXYupHA6plOuIbnS45LSQQs5qxEzV9pNNk64XpNOx6Z2LOrgvddN2n23Ik1wM41v4Qq9pyI-g9ynngiY3uCpDCMusVBQchVhgYhgIdFqTUUEcx8KHiksoigMeeYJ_xzUpvlUXgAnUowihSkyfL7cY9w0xsFI_-JCURbKBoCfGKZZ1VHcEFuMU-tZQJoa2FMDe1rB3gAPXyKzsp3GX5PrBvKNiSXal7_cvwN7_eRlkA6e4ucrsG8eVabqXYNaMV_KG7CbrYrRYn5r180H6mjAKQ |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Asymptotically+Optimal+Coded+Distributed+Computing+via+Combinatorial+Designs&rft.jtitle=IEEE%2FACM+transactions+on+networking&rft.au=Cheng%2C+Minquan&rft.au=Wu%2C+Youlong&rft.au=Li%2C+Xianxian&rft.au=Wu%2C+Dianhua&rft.date=2024-08-01&rft.pub=IEEE&rft.issn=1063-6692&rft.volume=32&rft.issue=4&rft.spage=3018&rft.epage=3033&rft_id=info:doi/10.1109%2FTNET.2024.3372698&rft.externalDocID=10462938 |
| thumbnail_l | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=1063-6692&client=summon |
| thumbnail_m | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=1063-6692&client=summon |
| thumbnail_s | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=1063-6692&client=summon |