A Fundamental Tradeoff Between Computation and Communication in Distributed Computing

How can we optimally trade extra computing power to reduce the communication load in distributed computing? We answer this question by characterizing a fundamental tradeoff between computation and communication in distributed computing, i.e., the two are inversely proportional to each other. More sp...

Celý popis

Uloženo v:
Podrobná bibliografie
Vydáno v:IEEE transactions on information theory Ročník 64; číslo 1; s. 109 - 128
Hlavní autoři: Songze Li, Maddah-Ali, Mohammad Ali, Qian Yu, Avestimehr, A. Salman
Médium: Journal Article
Jazyk:angličtina
Vydáno: New York IEEE 01.01.2018
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Témata:
Benchmark testing
coded multicasting
coded TeraSort
Coding
Communication
Computation
computation-communication tradeoff
Computer networks
Distributed computing
Distributed databases
Distributed processing
Electrical engineering
Electronic mail
Encoding
Information theory
Load distribution (forces)
Lower bounds
MapReduce
Nodes
Optimization
Stress concentration
Tradeoffs
ISSN:0018-9448, 1557-9654
On-line přístup:Získat plný text
Tagy: Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
Popis
Shrnutí:How can we optimally trade extra computing power to reduce the communication load in distributed computing? We answer this question by characterizing a fundamental tradeoff between computation and communication in distributed computing, i.e., the two are inversely proportional to each other. More specifically, a general distributed computing framework, motivated by commonly used structures like MapReduce, is considered, where the overall computation is decomposed into computing a set of "Map" and "Reduce" functions distributedly across multiple computing nodes. A coded scheme, named "coded distributed computing" (CDC), is proposed to demonstrate that increasing the computation load of the Map functions by a factor of r (i.e., evaluating each function at r carefully chosen nodes) can create novel coding opportunities that reduce the communication load by the same factor. An information-theoretic lower bound on the communication load is also provided, which matches the communication load achieved by the CDC scheme. As a result, the optimal computation-communication tradeoff in distributed computing is exactly characterized. Finally, the coding techniques of CDC is applied to the Hadoop TeraSort benchmark to develop a novel CodedTeraSort algorithm, which is empirically demonstrated to speed up the overall job execution by 1.97× -3.39×, for typical settings of interest.
Bibliografie:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:0018-9448
1557-9654
DOI:10.1109/TIT.2017.2756959