TACC: Topology-Aware Coded Computing for Distributed Graph Processing

This article proposes a coded distributed graph processing framework to alleviate the communication bottleneck in large-scale distributed graph processing. In particular, we propose a topology-aware coded computing (TACC) algorithm that has two novel salient features: (i) a topology-aware graph allo...

Celý popis

Uložené v:
Podrobná bibliografia
Vydané v:IEEE transactions on signal and information processing over networks Ročník 6; s. 508 - 525
Hlavní autori: Guler, Basak, Avestimehr, A. Salman, Ortega, Antonio
Médium: Journal Article
Jazyk:English
Vydavateľské údaje: Piscataway IEEE 2020
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Predmet:
Algorithms
Cloud computing
Communication
Computational modeling
Distributed computing
Encoding
graph signal filtering
large-scale graph processing
Load modeling
Multicast communication
Network topology
Reduction
Search engines
Sparse matrices
Stress concentration
Topology
ISSN:2373-776X, 2373-7778
On-line prístup:Získať plný text
Tagy: Pridať tag
Žiadne tagy, Buďte prvý, kto otaguje tento záznam!
Popis
Shrnutí:This article proposes a coded distributed graph processing framework to alleviate the communication bottleneck in large-scale distributed graph processing. In particular, we propose a topology-aware coded computing (TACC) algorithm that has two novel salient features: (i) a topology-aware graph allocation strategy, and (ii) a coded aggregation scheme that combines the intermediate computations for graph processing while constructing coded messages. The proposed setup results in a trade-off between computation and communication, in that increasing the computation load at the distributed parties can in turn reduce the communication load. We demonstrate the effectiveness of the TACC algorithm by comparing the communication load with existing setups on both Erdös-Rényi and Barabási-Albert type random graphs, as well as real-world Google web graph for PageRank computations. In particular, we show that the proposed coding strategy can lead to up to <inline-formula><tex-math notation="LaTeX">82\%</tex-math></inline-formula> reduction in communication load and up to <inline-formula><tex-math notation="LaTeX">46\%</tex-math></inline-formula> reduction in overall execution time, when compared to the state-of-the-art and implemented on the Amazon EC2 cloud compute platform.
Bibliografia:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:2373-776X
2373-7778
DOI:10.1109/TSIPN.2020.2998223