A Dynamic Sliding Window Based Tensor Communication Scheduling Framework for Distributed Deep Learning

Simultaneous tensor communication can effectively improve the scalability of distributed deep learning on large clusters. However, a fixed number of tensor blocks communicated concurrently violates the priority-based scheduling strategy and cannot minimize communication overheads. In this paper, we...

Full description

Saved in:

Bibliographic Details
Published in:	IEEE transactions on network science and engineering Vol. 12; no. 2; pp. 1080 - 1095
Main Authors:	Gao, Yunqi, Hu, Bing, Mashhadi, Mahdi Boloursaz, Wang, Wei, Tafazolli, Rahim, Debbah, Merouane
Format:	Journal Article
Language:	English
Published:	Piscataway IEEE 01.03.2025 The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subjects:	Artificial neural networks Clusters Communication communication scheduling Computational modeling Computer architecture data parallelism Deep learning Distributed deep learning Dynamic scheduling Energy consumption generative pre-trained transformer (GPT) Greedy algorithms Mathematical models Parallel processing Priority scheduling Processor scheduling Sliding tensor partitioning Tensors Training
ISSN:	2327-4697, 2334-329X
Online Access:	Get full text
Tags:	Add Tag No Tags, Be the first to tag this record!

Be the first to leave a comment!