Asymptotically Optimal Coded Distributed Computing via Combinatorial Designs

Coded distributed computing (CDC) introduced by Li et al. can greatly reduce the communication load for MapReduce computing systems. In the cascaded CDC with <inline-formula> <tex-math notation="LaTeX">K </tex-math></inline-formula> workers, <inline-formula> &...

Full description

Saved in:
Bibliographic Details
Published in:IEEE/ACM transactions on networking Vol. 32; no. 4; pp. 3018 - 3033
Main Authors: Cheng, Minquan, Wu, Youlong, Li, Xianxian, Wu, Dianhua
Format: Journal Article
Language:English
Published: IEEE 01.08.2024
Subjects:
asymptotically optimal
Coded distributed computing
Costs
Distributed computing
Encoding
Symbols
t-design
t-GDD
Task analysis
Technological innovation
Vectors
ISSN:1063-6692, 1558-2566
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Coded distributed computing (CDC) introduced by Li et al. can greatly reduce the communication load for MapReduce computing systems. In the cascaded CDC with <inline-formula> <tex-math notation="LaTeX">K </tex-math></inline-formula> workers, <inline-formula> <tex-math notation="LaTeX">N </tex-math></inline-formula> input files and <inline-formula> <tex-math notation="LaTeX">Q </tex-math></inline-formula> output functions, each input file will be mapped by <inline-formula> <tex-math notation="LaTeX">r </tex-math></inline-formula> workers and each output function will be computed by <inline-formula> <tex-math notation="LaTeX">s </tex-math></inline-formula> workers such that coding techniques can be applied to create multicast opportunities. The main drawback of most existing CDC schemes is that they require the original data to be split into a large number of input files that grows exponentially with <inline-formula> <tex-math notation="LaTeX">K </tex-math></inline-formula>, which would significantly increase the coding complexity and degrade the system performance. In this paper, we first use a classical combinatorial structure <inline-formula> <tex-math notation="LaTeX">t </tex-math></inline-formula>-design, for any integer <inline-formula> <tex-math notation="LaTeX">t\geq 2 </tex-math></inline-formula>, to develop a low-complexity and communication-efficient CDC with <inline-formula> <tex-math notation="LaTeX">r=s </tex-math></inline-formula>. Our scheme has much smaller <inline-formula> <tex-math notation="LaTeX">N </tex-math></inline-formula> and <inline-formula> <tex-math notation="LaTeX">Q </tex-math></inline-formula> than the existing schemes under the same parameters <inline-formula> <tex-math notation="LaTeX">K </tex-math></inline-formula>, <inline-formula> <tex-math notation="LaTeX">r </tex-math></inline-formula>, and <inline-formula> <tex-math notation="LaTeX">s </tex-math></inline-formula>; and achieves smaller communication loads compared with the state-of-the-art schemes when <inline-formula> <tex-math notation="LaTeX">K </tex-math></inline-formula> is relatively large. Remarkably, unlike the previous schemes that realize on large operation fields, our scheme operates in one-shot communication on the minimum binary field <inline-formula> <tex-math notation="LaTeX">\mathbb {F}_{2} </tex-math></inline-formula>. With a derived lower bound on the communication load under one-shot linear delivery, we show that the <inline-formula> <tex-math notation="LaTeX">t </tex-math></inline-formula>-design scheme is asymptotically optimal. Furthermore, we show that our construction method can incorporate the other combinatorial structures that have a similar property to <inline-formula> <tex-math notation="LaTeX">t </tex-math></inline-formula>-design. For instance, we use <inline-formula> <tex-math notation="LaTeX">t </tex-math></inline-formula>-GDD to obtain another one-shot asymptotically optimal CDC scheme over <inline-formula> <tex-math notation="LaTeX">\mathbb {F}_{2} </tex-math></inline-formula> that has different parameters from <inline-formula> <tex-math notation="LaTeX">t </tex-math></inline-formula>-design. Finally, we show that our construction method can also be used to construct CDC schemes with <inline-formula> <tex-math notation="LaTeX">r\neq s </tex-math></inline-formula> that have small file number and output function number.
ISSN:1063-6692
1558-2566
DOI:10.1109/TNET.2024.3372698